Oct 31 19:16:47.333734 ip-10-0-138-35 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 19:16:47.333747 ip-10-0-138-35 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 19:16:47.333755 ip-10-0-138-35 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 19:16:47.334061 ip-10-0-138-35 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 19:16:57.582845 ip-10-0-138-35 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 19:16:57.582863 ip-10-0-138-35 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3e7d9fa8dd644210a9a1507e7fa4fe82 --
Oct 31 19:18:53.501887 ip-10-0-138-35 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 19:18:53.857670 ip-10-0-138-35 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:18:53.857670 ip-10-0-138-35 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 19:18:53.857670 ip-10-0-138-35 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:18:53.857670 ip-10-0-138-35 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:18:53.861739 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861658    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.861739 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861739    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861743    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861747    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861750    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861753    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861756    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861758    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861761    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861764    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861766    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861769    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861772    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861774    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861777    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861781    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861783    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861786    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861789    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861791    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861794    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.861810 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861796    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861799    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861802    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861805    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861808    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861811    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861814    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861817    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861819    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861822    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861825    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861828    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861831    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861833    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861836    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861838    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861841    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861843    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861846    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861848    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.862276 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861851    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861853    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861856    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861861    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861863    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861866    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861868    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861871    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861873    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861876    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861879    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861882    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861884    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861888    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861891    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861894    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861896    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861899    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861901    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.862766 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861904    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861906    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861909    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861912    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861914    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861917    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861921    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861925    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861929    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861931    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861934    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861937    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861940    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861943    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861945    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861948    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861950    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861953    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861956    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.863236 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861958    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861961    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861963    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861966    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861970    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861974    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861977    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861981    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861984    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861987    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861990    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861993    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861995    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.861998    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862000    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862003    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862005    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862008    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862011    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.863697 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862013    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862016    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862018    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862021    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862024    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862026    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862029    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862832    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862840    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862843    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862846    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862849    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862852    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862856    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862861    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862863    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862866    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862869    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862872    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.864150 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862875    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862879    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862882    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862885    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862888    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862890    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862893    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862896    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862898    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862901    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862903    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862906    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862909    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862911    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862914    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862916    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862919    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862922    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862924    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862926    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.864614 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862930    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862933    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862936    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862938    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862941    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862943    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862946    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862948    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862951    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862955    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862958    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862962    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862965    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862969    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862972    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862974    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862977    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862979    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862982    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.865090 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862985    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862987    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862990    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862992    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862995    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.862997    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863000    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863002    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863005    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863008    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863010    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863014    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863016    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863019    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863022    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863024    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863027    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863029    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863032    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863035    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.865557 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863038    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863040    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863043    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863045    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863048    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863051    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863053    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863056    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863059    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863061    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863064    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863066    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863069    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863072    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863075    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863078    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863080    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863083    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863085    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863088    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.866030 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863091    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863094    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863096    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863099    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863102    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863104    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863107    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863110    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863112    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863115    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863117    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863121    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863123    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863126    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863128    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863645    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863652    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863656    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863659    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863662    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.866527 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863665    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863667    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863670    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863673    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863676    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863678    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863681    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863683    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863686    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863688    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863692    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863694    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863697    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863700    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863703    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863705    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863708    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863712    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863716    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863720    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.867022 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863723    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863726    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863728    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863731    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863734    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863737    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863739    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863742    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863745    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863747    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863750    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863753    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863755    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863758    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863761    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863763    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863766    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863768    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863771    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863773    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.867489 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863776    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863779    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863782    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863784    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863787    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863790    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863792    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863795    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863798    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863801    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863803    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863806    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863809    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863811    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863814    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863817    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863819    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863822    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863824    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.868016 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863827    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863830    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863832    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863835    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863838    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863840    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863843    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863845    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863848    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863850    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863853    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863856    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863858    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863861    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863864    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863866    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863869    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863871    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863874    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863876    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.868449 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863879    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863881    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863884    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863886    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863889    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863891    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863894    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863897    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863900    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863902    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863905    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863907    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863910    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863912    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863915    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863918    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863921    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863923    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863926    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.868979 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863932    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863937    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.863941    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864751    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864760    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864767    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864772    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864777    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864781    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864790    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864795    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864799    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864802    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864806    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864809    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864812    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864815    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864819    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864822    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864825    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864832    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864837    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864840    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 19:18:53.869438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864844    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864847    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864850    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864859    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864864    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864867    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864871    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864874    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864877    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864881    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864884    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864888    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864893    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864896    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864899    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864901    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864905    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864908    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864912    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864915    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864918    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864922    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864925    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864929    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 19:18:53.870007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864932    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864935    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864938    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864942    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864945    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864948    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864951    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864954    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864957    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864960    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864964    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864967    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864972    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864975    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864978    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864981    2577 flags.go:64] FLAG: --help="false"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864984    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864988    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864991    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864994    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.864998    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865001    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865004    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 19:18:53.870675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865007    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865010    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865013    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865016    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865019    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865022    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865025    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865028    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865031    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865034    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865036    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865039    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865042    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865046    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865051    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865054    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865058    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865062    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865065    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865068    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865071    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865075    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865080    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865083    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865087    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 19:18:53.871225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865090    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865094    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865097    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865100    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865104    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865107    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865110    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865117    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865120    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865123    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865126    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865129    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865131    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865135    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865137    2577 flags.go:64] FLAG: --port="10250"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865140    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865143    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01d9b77015ae7a388"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865146    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865149    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865152    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865156    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865159    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865163    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865166    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865169    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865173    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 19:18:53.871902 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865176    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865179    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865185    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865188    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865191    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865194    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865197    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865201    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865204    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865207    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865210    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865213    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865216    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865219    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865222    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865225    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865228    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865231    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865234    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865239    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865242    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865245    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865252    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865255    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865258    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 19:18:53.872531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865260    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865263    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865266    2577 flags.go:64] FLAG: --v="2"
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865272    2577 flags.go:64] FLAG: --version="false"
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865282    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865287    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865291    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865431    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865434    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865438    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865441    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865444    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865446    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865449    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865453    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865456    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865458    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865461    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865464    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865467    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865470    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.873130 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865472    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865475    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865477    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865480    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865483    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865485    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865488    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865490    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865508    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865512    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865515    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865518    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865521    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865524    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865527    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865530    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865533    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865535    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865538    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865552    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.873665 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865557    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865560    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865562    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865565    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865568    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865571    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865574    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865577    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865579    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865582    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865585    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865587    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865590    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865592    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865595    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865598    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865600    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865603    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865605    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865608    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.874285 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865610    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865613    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865615    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865618    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865621    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865623    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865626    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865628    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865631    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865633    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865636    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865639    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865642    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865645    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865648    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865650    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865653    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865656    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865659    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.875044 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865662    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865664    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865667    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865669    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865672    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865675    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865678    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865680    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865683    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865686    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865688    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865691    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865693    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865697    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865700    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865703    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865706    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865709    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.875568 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865713    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865716    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865718    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865721    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865724    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865726    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865730    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865733    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865736    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865738    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865741    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865744    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865747    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865751    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.865778    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:18:53.876028 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865815    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865819    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865822    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865824    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865827    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865830    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865832    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865835    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865838    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865841    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865843    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865846    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865848    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865851    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865853    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865856    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865859    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865861    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865864    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865867    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.876400 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865870    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865872    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865875    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865879    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865882    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865885    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865887    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865890    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865892    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865897    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865899    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865902    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865904    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865907    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865910    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865912    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865915    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865917    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865919    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865922    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.876893 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865925    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865927    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865930    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865932    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865935    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865938    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865940    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865943    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865946    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865948    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865951    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865953    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865956    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865958    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865961    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865965    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865968    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865971    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865974    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.877372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865976    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865979    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865983    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865985    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865988    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865990    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865993    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865995    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.865998    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866001    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866003    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866006    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866008    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866011    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866015    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866018    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866021    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866023    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866026    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.877832 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866028    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866031    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866034    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866037    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866039    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866042    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866045    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866047    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866050    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866053    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866056    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866058    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866061    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866064    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866068    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866073    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866076    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866079    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866082    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.878268 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866084    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866087    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866090    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866092    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866095    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866097    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866101    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866104    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.866106    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.872993    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.873014    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.873030    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.873035    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873097    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873102    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873106    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873109    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873111    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873114    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.878798 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873117    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873119    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873122    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873125    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873128    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873130    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873134    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873137    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873140    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873143    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873147    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873149    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873152    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873155    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873157    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873160    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873163    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873165    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873168    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873170    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.879316 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873173    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873176    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873179    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873182    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873185    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873188    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873190    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873193    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873196    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873198    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873201    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873203    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873206    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873208    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873211    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873214    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873216    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873219    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873222    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873225    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.879831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873228    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873231    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873233    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873236    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873238    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873241    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873244    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873247    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873250    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873252    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873255    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873257    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873260    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873263    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873266    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873268    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873271    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873274    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873284    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.880358 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873287    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873289    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873292    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873295    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873298    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873300    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873303    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873305    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873308    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873310    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873313    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873315    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873318    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873321    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873324    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873326    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873329    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873333    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873336    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.881182 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873339    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873343    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873345    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873348    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873350    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873353    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873356    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873358    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873361    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873363    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873367    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873371    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873374    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873377    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873379    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873383    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873385    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873388    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873390    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873393    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.881899 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873395    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.873398    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.873430    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.874986    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875002    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875008    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875013    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875019    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875026    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875029    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875035    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875039    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875044    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875048    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875054    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875058    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.882372 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875063    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875067    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875072    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875076    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875081    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875089    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875094    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875098    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875155    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875279    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875284    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875288    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875291    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875294    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875297    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875299    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875302    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875305    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875307    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875310    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.882879 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875313    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875316    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875320    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875325    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875329    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875337    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875341    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875344    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875347    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875349    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875352    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875355    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875357    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875360    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875363    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875366    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875369    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875372    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.883488 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875375    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875379    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875384    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875388    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875392    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875395    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875398    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875401    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875403    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875406    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875409    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875411    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875414    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875416    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875419    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875421    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875424    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875427    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875429    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.884008 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875432    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875435    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875438    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875441    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875443    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875446    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875449    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875452    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875454    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875457    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875459    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875462    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875465    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875467    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875470    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875474    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875477    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875479    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875482    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875485    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.884462 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875488    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875490    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875492    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875495    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875498    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875500    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875503    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875505    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875508    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875510    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875513    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875516    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875519    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875521    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875524    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875527    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.884976 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.875578    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875621    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875626    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875629    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875632    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875635    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875638    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875640    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875643    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875646    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875649    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875652    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875654    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875657    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875660    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875663    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875666    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875668    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875671    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875673    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:53.885380 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875676    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875678    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875681    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875683    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875686    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875688    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875691    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875694    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875696    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875699    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875701    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875704    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875706    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875709    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875712    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875714    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875717    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875719    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875722    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875724    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:53.885857 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875727    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875730    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875732    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875736    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875738    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875741    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875743    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875746    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875748    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875750    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875753    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875755    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875758    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875760    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875763    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875765    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875768    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875771    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875773    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875776    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:53.886353 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875778    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875781    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875784    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875788    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875794    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875796    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875799    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875802    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875805    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875807    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875810    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875813    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875816    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875818    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875821    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875825    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875828    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875830    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:53.886831 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875833    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875837    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875841    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875844    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875846    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875849    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875852    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875854    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875857    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875860    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875862    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875864    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875867    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875869    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875872    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875874    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875877    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875879    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875883    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:53.887247 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875887    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875890    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875893    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875895    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875898    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875902    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875905    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875907    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875910    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:53.875913    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.875998    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.878163    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.878985    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.879090    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.879803    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.880862    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:18:53.887710 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.883507    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:18:53.893136 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.893120    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 19:18:53.896711 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.896698    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 19:18:53.897607 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.897583    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 19:18:53.902313 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.902290    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a06d57c5-f82c-423b-ae6b-871f70db6464:/dev/nvme0n1p3 feb6b71b-649b-43d0-b9c8-8951155b333a:/dev/nvme0n1p4]
Oct 31 19:18:53.902400 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.902311    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 19:18:53.908463 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.908335    2577 manager.go:217] Machine: {Timestamp:2026-10-31 19:18:53.9073876 +0000 UTC m=+0.301226266 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108049 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287c06a8d3af57ea3be40edfbf38e0 SystemUUID:ec287c06-a8d3-af57-ea3b-e40edfbf38e0 BootID:3e7d9fa8-dd64-4210-a9a1-507e7fa4fe82 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:18:a7:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:18:a7:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:4d:59:cc:64:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 19:18:53.908463 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.908451    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 19:18:53.908603 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.908531    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 19:18:53.911424 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.911397    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 19:18:53.911602 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.911425    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-35.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 19:18:53.911659 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.911605    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 19:18:53.911659 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.911615    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 19:18:53.911659 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.911656    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:18:53.911756 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.911675    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:18:53.912761 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.912751    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 19:18:53.915945 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.915928    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 19:18:53.916969 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.916957    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 19:18:53.917088 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917079    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 19:18:53.917122 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917111    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 19:18:53.917122 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917120    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 19:18:53.917174 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917130    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 19:18:53.917174 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917141    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 19:18:53.917174 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917149    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 19:18:53.917507 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.917493    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:18:53.919450 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.919428    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 19:18:53.925839 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.925823    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:18:53.926778 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926767    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 19:18:53.926826 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926796    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 19:18:53.926945 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926935    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 19:18:53.926976 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926954    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 19:18:53.926976 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926964    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 19:18:53.926976 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926973    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 19:18:53.927056 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.926981    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 19:18:53.927056 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927010    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 19:18:53.927580 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927570    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 19:18:53.927615 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927580    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 19:18:53.927615 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927587    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 19:18:53.927615 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927594    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 19:18:53.927615 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927603    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 19:18:53.927733 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.927618    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 19:18:53.928250 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.928241    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 19:18:53.928250 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.928250    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 19:18:53.932181 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.932165    2577 server.go:1263] "Started kubelet"
Oct 31 19:18:53.932655 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.932572    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 19:18:53.932861 ip-10-0-138-35 systemd[1]: Started Kubernetes Kubelet.
Oct 31 19:18:53.933095 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.932891    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-35.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 19:18:53.933095 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.932949    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 19:18:53.933095 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.933016    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 19:18:53.935913 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.935894    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 19:18:53.937692 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.937643    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 19:18:53.938693 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.938670    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 19:18:53.938987 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.938010    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.us-east-2.compute.internal.18e3b3678bce321d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:53.932139037 +0000 UTC m=+0.325977705,LastTimestamp:2026-10-31 19:18:53.932139037 +0000 UTC m=+0.325977705,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.us-east-2.compute.internal,}"
Oct 31 19:18:53.943106 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.943081    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 19:18:53.943679 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.943661    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 19:18:53.944744 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.944491    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:53.945246 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945228    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 19:18:53.945246 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945232    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 19:18:53.945393 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945258    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 19:18:53.945393 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945346    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 19:18:53.945393 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945352    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 19:18:53.945629 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945613    2577 factory.go:55] Registering systemd factory
Oct 31 19:18:53.945667 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945633    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 19:18:53.945872 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945857    2577 factory.go:153] Registering CRI-O factory
Oct 31 19:18:53.945915 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945875    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 19:18:53.945958 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945928    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 19:18:53.945958 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945945    2577 factory.go:103] Registering Raw factory
Oct 31 19:18:53.946018 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.945963    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 19:18:53.946873 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.946857    2577 manager.go:319] Starting recovery of all containers
Oct 31 19:18:53.948036 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.947875    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-35.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 19:18:53.948036 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.947944    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-35.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 19:18:53.948995 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.948976    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdn64"
Oct 31 19:18:53.951712 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.951681    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 19:18:53.954633 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.954608    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 19:18:53.958373 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.958237    2577 manager.go:324] Recovery completed
Oct 31 19:18:53.958434 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.958408    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdn64"
Oct 31 19:18:53.965989 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.965968    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:53.967826 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.967802    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:53.967957 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.967844    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:53.967957 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.967860    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:53.968479 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.968463    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 19:18:53.968479 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.968478    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 19:18:53.968597 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.968466    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:53.968597 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.968497    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 19:18:53.969217 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.969203    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:53.969270 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.969230    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:53.969270 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:53.969240    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:53.971422 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:53.971348    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.us-east-2.compute.internal.18e3b3678deebd8d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-35.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:53.967826317 +0000 UTC m=+0.361664986,LastTimestamp:2026-10-31 19:18:53.967826317 +0000 UTC m=+0.361664986,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.us-east-2.compute.internal,}"
Oct 31 19:18:54.044987 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.044951    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.104557 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.104520    2577 policy_none.go:50] "Start"
Oct 31 19:18:54.104662 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.104583    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 19:18:54.104662 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.104625    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 19:18:54.106826 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.106812    2577 policy_none.go:44] "Start"
Oct 31 19:18:54.138328 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.138139    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 19:18:54.142716 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.142703    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 19:18:54.142789 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.142737    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 19:18:54.142789 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.142749    2577 server.go:87] "Starting device plugin registration server"
Oct 31 19:18:54.143011 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.142998    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 19:18:54.143060 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.143019    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 19:18:54.143102 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.143089    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 19:18:54.143187 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.143172    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 19:18:54.143228 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.143187    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 19:18:54.143954 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.143929    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 19:18:54.144047 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.143991    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.154704 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.154679    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 19:18:54.155868 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.155848    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 19:18:54.155868 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.155869    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 19:18:54.156000 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.155892    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 19:18:54.156000 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.155931    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 19:18:54.156090 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.156010    2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-35.us-east-2.compute.internal\" not found" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.165923 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.165905    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:54.244065 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.244034    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:54.250006 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.249984    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:54.250119 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.250017    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:54.250119 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.250027    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:54.250119 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.250074    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.256002 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.255981    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"]
Oct 31 19:18:54.256055 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.256045    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:54.256949 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.256933    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:54.257041 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.256960    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:54.257041 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.256970    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:54.259341 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.259328    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:54.259496 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.259479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.259574 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.259520    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:54.260132 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.260117    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:54.260204 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.260133    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:54.260204 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.260151    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:54.260204 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.260156    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:54.260204 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.260167    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:54.260204 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.260170    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:54.262112 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.262092    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.262164 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.262123    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-35.us-east-2.compute.internal\": node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.262403 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.262391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.262447 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.262413    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:54.263105 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.263091    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:54.263186 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.263118    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:54.263186 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.263133    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:54.290266 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.290245    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.292669 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.292653    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.303973 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.303953    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.346537 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.346504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.346673 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.346558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.346673 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.346649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1d262f6c678a43aaab0402c233dbed3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"a1d262f6c678a43aaab0402c233dbed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.404331 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.404308    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.447707 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.447676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.447824 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.447711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.447824 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.447728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1d262f6c678a43aaab0402c233dbed3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"a1d262f6c678a43aaab0402c233dbed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.447824 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.447791    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1d262f6c678a43aaab0402c233dbed3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"a1d262f6c678a43aaab0402c233dbed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.447824 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.447793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.447824 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.447799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.505085 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.505058    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.595170 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.595135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.597286 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.597261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.605483 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.605466    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.705675 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.705633    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.810537 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.810493    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.880652 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.880615    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 19:18:54.881152 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.880809    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 19:18:54.911204 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:54.911177    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:18:54.936793 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.936733    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:54.943634 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.943613    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 19:18:54.945269 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.945251    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.959083 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.959056    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:18:54.960869 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.960852    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:18:54.962224 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.962209    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 19:18:54.967263 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.966533    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:13:54 +0000 UTC" deadline="2028-05-08 05:21:34.681563476 +0000 UTC"
Oct 31 19:18:54.967511 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.967267    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13306h2m39.714301244s"
Oct 31 19:18:54.971698 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:54.971676    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:18:55.041328 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.041297    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:55.051812 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.051787    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vdrdr"
Oct 31 19:18:55.067282 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.067256    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vdrdr"
Oct 31 19:18:55.317215 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:55.317177    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12c7c191df1c0e91dfcc512de902352.slice/crio-c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f WatchSource:0}: Error finding container c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f: Status 404 returned error can't find the container with id c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f
Oct 31 19:18:55.319295 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:55.319262    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d262f6c678a43aaab0402c233dbed3.slice/crio-1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e WatchSource:0}: Error finding container 1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e: Status 404 returned error can't find the container with id 1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e
Oct 31 19:18:55.323153 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.323132    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:18:55.324144 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.324127    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:55.918724 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.918692    2577 apiserver.go:52] "Watching apiserver"
Oct 31 19:18:55.937357 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.937313    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 19:18:55.938061 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.938026    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9xhcv","kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal","kube-system/konnectivity-agent-vgtp6","openshift-multus/multus-additional-cni-plugins-qswht","openshift-ovn-kubernetes/ovnkube-node-9njm5","kube-system/global-pull-secret-syncer-bz76x","openshift-cluster-node-tuning-operator/tuned-7w6mv","openshift-dns/node-resolver-w6g8x","openshift-image-registry/node-ca-6qz8h","openshift-multus/multus-jv8rs","openshift-network-operator/iptables-alerter-9wm8m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2","openshift-multus/network-metrics-daemon-bfhh2"]
Oct 31 19:18:55.941769 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.940855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.943558 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.943531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:55.943684 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:55.943665    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:18:55.945195 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.945170    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 19:18:55.945462 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.945442    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 19:18:55.945562 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.945544    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w5n5s\""
Oct 31 19:18:55.945737 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.945718    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 19:18:55.945800 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.945719    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 19:18:55.949076 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.949056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:55.955531    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-kubelet\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-socket-dir-parent\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-bin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-daemon-config\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtq2\" (UniqueName: \"kubernetes.io/projected/a989c545-d91d-4ca2-9dea-0637a462d204-kube-api-access-kjtq2\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-hostroot\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.955992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-multus-certs\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmdg\" (UniqueName: \"kubernetes.io/projected/22143d8e-6fcb-4563-8989-a29ddefbc053-kube-api-access-ttmdg\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-registration-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-device-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-cnibin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-k8s-cni-cncf-io\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.956978 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-netns\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-multus\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-conf-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-etc-kubernetes\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-sys-fs\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-system-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-cni-binary-copy\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-socket-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.957894 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.956463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7q4\" (UniqueName: \"kubernetes.io/projected/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kube-api-access-kk7q4\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:55.963277 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.963236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:55.967726 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.966746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:55.969942 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.969630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:55.976389 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.976002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:55.978696 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.978675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:55.982126 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.982098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:55.985401 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.985377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:55.987676 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:55.987420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:55.987676 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:55.987476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:18:56.046916 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.046885    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 19:18:56.053739 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.053715    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 19:18:56.054549 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.054530    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:18:56.054763 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.054748    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 19:18:56.054912 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.054890    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 19:18:56.055011 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.054996    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:18:56.055141 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055128    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 19:18:56.055308 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055295    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 19:18:56.055442 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055429    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 19:18:56.055590 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055577    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 19:18:56.055744 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055731    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 19:18:56.055920 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055899    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 19:18:56.056009 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055946    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 19:18:56.056069 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056020    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 19:18:56.056128 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056089    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 19:18:56.056183 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056142    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 19:18:56.056238 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056183    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 19:18:56.056238 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056224    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 19:18:56.056338 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056265    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cs8fb\""
Oct 31 19:18:56.056338 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056304    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 19:18:56.056468 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056345    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 19:18:56.056468 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056367    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 19:18:56.056468 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056402    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 19:18:56.056468 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056430    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcmmt\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056471    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rs77w\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056502    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwg2q\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056545    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7nlh\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056544    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056658    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbgxr\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056677    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbst6\""
Oct 31 19:18:56.056741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.055015    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-slash\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-konnectivity-ca\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92bf693-c264-4e52-84a5-045804009257-tmp-dir\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056931    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gfbm\""
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.056951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-cnibin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.057019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-k8s-cni-cncf-io\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-sys-fs\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-sys-fs\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-k8s-cni-cncf-io\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-etc-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-cnibin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysconfig\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgx2q\" (UniqueName: \"kubernetes.io/projected/d92bf693-c264-4e52-84a5-045804009257-kube-api-access-bgx2q\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a237adff-6089-4d22-993f-95a9869a9771-host\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.057333 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-dbus\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-node-log\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-netd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-env-overrides\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-systemd-units\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-ovn\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-script-lib\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-tuned\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d92bf693-c264-4e52-84a5-045804009257-hosts-file\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-conf\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-systemd\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-conf-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.057831 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-cni-binary-copy\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0312583f-6efa-4d8f-8921-d9deb835d69f-host-slash\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.057989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-run\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-daemon-config\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-hostroot\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-multus-certs\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmdg\" (UniqueName: \"kubernetes.io/projected/22143d8e-6fcb-4563-8989-a29ddefbc053-kube-api-access-ttmdg\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-registration-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-agent-certs\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-netns\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-etc-kubernetes\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7q4\" (UniqueName: \"kubernetes.io/projected/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kube-api-access-kk7q4\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.058465 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-log-socket\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-kubernetes\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058495    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-cni-binary-copy\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkmw9\" (UniqueName: \"kubernetes.io/projected/a237adff-6089-4d22-993f-95a9869a9771-kube-api-access-nkmw9\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-kubelet\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058765    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-hostroot\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-kubelet\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtq2\" (UniqueName: \"kubernetes.io/projected/a989c545-d91d-4ca2-9dea-0637a462d204-kube-api-access-kjtq2\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058848    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-multus-certs\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-systemd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-sys\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-registration-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058962    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-etc-kubernetes\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-netns\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.058994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7sgm\" (UniqueName: \"kubernetes.io/projected/d48cdb47-67b2-47ec-9433-d831d8393f53-kube-api-access-w7sgm\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.059281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-device-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-lib-modules\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-var-lib-kubelet\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-multus\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cnibin\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-bin\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-config\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghc9v\" (UniqueName: \"kubernetes.io/projected/0312583f-6efa-4d8f-8921-d9deb835d69f-kube-api-access-ghc9v\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-host\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a237adff-6089-4d22-993f-95a9869a9771-serviceca\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-system-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-socket-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-system-cni-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059708    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-device-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.060113 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-var-lib-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059768    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059767    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-kubelet\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059791    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-multus\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.059863    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.059910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-system-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.059931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:56.559906425 +0000 UTC m=+2.953745097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovn-node-metrics-cert\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfzk\" (UniqueName: \"kubernetes.io/projected/c9647c81-f8aa-470d-a5c7-02897a93fa31-kube-api-access-6bfzk\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-tmp\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060184    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-conf-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-socket-dir-parent\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-bin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-netns\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0312583f-6efa-4d8f-8921-d9deb835d69f-iptables-alerter-script\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.060964 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-socket-dir-parent\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-modprobe-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-kubelet-config\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64r5\" (UniqueName: \"kubernetes.io/projected/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-kube-api-access-x64r5\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.060530    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-bin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.061024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-daemon-config\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.061848 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.061106    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-socket-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.070725 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.070670    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:55 +0000 UTC" deadline="2028-05-12 14:48:03.356041743 +0000 UTC"
Oct 31 19:18:56.070725 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.070724    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13411h29m7.285321552s"
Oct 31 19:18:56.077679 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.077638    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 19:18:56.083275 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.083244    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtq2\" (UniqueName: \"kubernetes.io/projected/a989c545-d91d-4ca2-9dea-0637a462d204-kube-api-access-kjtq2\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:56.084164 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.084142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7q4\" (UniqueName: \"kubernetes.io/projected/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kube-api-access-kk7q4\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.085952 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.085919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmdg\" (UniqueName: \"kubernetes.io/projected/22143d8e-6fcb-4563-8989-a29ddefbc053-kube-api-access-ttmdg\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.160734 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-system-cni-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.160893 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-var-lib-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.160893 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovn-node-metrics-cert\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.160893 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfzk\" (UniqueName: \"kubernetes.io/projected/c9647c81-f8aa-470d-a5c7-02897a93fa31-kube-api-access-6bfzk\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.160893 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-tmp\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.160893 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-netns\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.160893 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0312583f-6efa-4d8f-8921-d9deb835d69f-iptables-alerter-script\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-modprobe-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.160991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-kubelet-config\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64r5\" (UniqueName: \"kubernetes.io/projected/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-kube-api-access-x64r5\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-slash\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-konnectivity-ca\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92bf693-c264-4e52-84a5-045804009257-tmp-dir\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-etc-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysconfig\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx2q\" (UniqueName: \"kubernetes.io/projected/d92bf693-c264-4e52-84a5-045804009257-kube-api-access-bgx2q\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a237adff-6089-4d22-993f-95a9869a9771-host\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.161213 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-dbus\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-node-log\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-netd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-env-overrides\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-systemd-units\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-ovn\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-script-lib\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-tuned\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d92bf693-c264-4e52-84a5-045804009257-hosts-file\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-conf\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-systemd\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0312583f-6efa-4d8f-8921-d9deb835d69f-host-slash\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-run\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.161762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-agent-certs\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-log-socket\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-kubernetes\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmw9\" (UniqueName: \"kubernetes.io/projected/a237adff-6089-4d22-993f-95a9869a9771-kube-api-access-nkmw9\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-kubelet\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-systemd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.161974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-sys\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.162000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7sgm\" (UniqueName: \"kubernetes.io/projected/d48cdb47-67b2-47ec-9433-d831d8393f53-kube-api-access-w7sgm\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.162028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.162060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-lib-modules\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.162095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-var-lib-kubelet\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.162122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.162239    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:56.162545 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.162299    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:56.662281177 +0000 UTC m=+3.056119833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.162797    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-conf\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163015    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163137    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-ovn\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cnibin\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-bin\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-config\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.163280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc9v\" (UniqueName: \"kubernetes.io/projected/0312583f-6efa-4d8f-8921-d9deb835d69f-kube-api-access-ghc9v\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.163665 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-host\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.163665 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a237adff-6089-4d22-993f-95a9869a9771-serviceca\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.163665 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163412    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-script-lib\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.163665 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163461    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-log-socket\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.163665 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-kubernetes\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.163928 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163820    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-kubelet\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.163928 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a237adff-6089-4d22-993f-95a9869a9771-serviceca\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.163928 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.163870    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.164076 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164002    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.164076 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164068    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-systemd\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.164182 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-systemd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.164182 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164117    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0312583f-6efa-4d8f-8921-d9deb835d69f-host-slash\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.164182 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164148    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-sys\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.164329 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164283    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-systemd-units\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.164478 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-env-overrides\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.164478 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164460    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.164663 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-netd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.164722 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-konnectivity-ca\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.164765 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.164746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-run\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165259    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-kubelet-config\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165297    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-slash\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-host\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-dbus\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-system-cni-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165942    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysconfig\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165961    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-node-log\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.165985    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cnibin\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166011    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-etc-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166033    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-bin\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-netns\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.166218 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166216    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-var-lib-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.167068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a237adff-6089-4d22-993f-95a9869a9771-host\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.167068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166274    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d92bf693-c264-4e52-84a5-045804009257-hosts-file\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.167068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166279    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.167068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166329    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-modprobe-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.167068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166420    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-var-lib-kubelet\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.167068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.166438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-lib-modules\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.168228 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.168171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"a1d262f6c678a43aaab0402c233dbed3","Type":"ContainerStarted","Data":"1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e"}
Oct 31 19:18:56.174519 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.169923    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"b12c7c191df1c0e91dfcc512de902352","Type":"ContainerStarted","Data":"c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f"}
Oct 31 19:18:56.175568 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.175539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92bf693-c264-4e52-84a5-045804009257-tmp-dir\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.175739 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.175712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-agent-certs\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.175841 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.175798    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0312583f-6efa-4d8f-8921-d9deb835d69f-iptables-alerter-script\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.175943 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.175911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.175986 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.175944    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-tuned\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.176442 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.176418    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.176718 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.176694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovn-node-metrics-cert\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.176890 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.176864    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-config\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.177887 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.177862    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-tmp\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.190283 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.189792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64r5\" (UniqueName: \"kubernetes.io/projected/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-kube-api-access-x64r5\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.191385 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.191359    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmw9\" (UniqueName: \"kubernetes.io/projected/a237adff-6089-4d22-993f-95a9869a9771-kube-api-access-nkmw9\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.194447 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.194409    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:56.194447 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.194442    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:56.194632 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.194459    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:56.194632 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.194530    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:18:56.694509876 +0000 UTC m=+3.088348552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:56.194632 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.194539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx2q\" (UniqueName: \"kubernetes.io/projected/d92bf693-c264-4e52-84a5-045804009257-kube-api-access-bgx2q\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.194632 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.194612    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7sgm\" (UniqueName: \"kubernetes.io/projected/d48cdb47-67b2-47ec-9433-d831d8393f53-kube-api-access-w7sgm\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.196246 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.196225    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc9v\" (UniqueName: \"kubernetes.io/projected/0312583f-6efa-4d8f-8921-d9deb835d69f-kube-api-access-ghc9v\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.197324 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.197302    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfzk\" (UniqueName: \"kubernetes.io/projected/c9647c81-f8aa-470d-a5c7-02897a93fa31-kube-api-access-6bfzk\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.260166 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.260133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jv8rs"
Oct 31 19:18:56.271475 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.271448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:18:56.282170 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.282138    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e5ab86_50ff_419a_a4da_2c19408eb9ae.slice/crio-52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9 WatchSource:0}: Error finding container 52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9: Status 404 returned error can't find the container with id 52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9
Oct 31 19:18:56.297431 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.297404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:18:56.309968 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.309937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:18:56.310319 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.310270    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ccae21_e8ff_4b3a_b613_7fdd52aeacbf.slice/crio-92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c WatchSource:0}: Error finding container 92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c: Status 404 returned error can't find the container with id 92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c
Oct 31 19:18:56.313043 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.312730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:18:56.319965 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.319943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:18:56.322320 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.321299    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0312583f_6efa_4d8f_8921_d9deb835d69f.slice/crio-9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47 WatchSource:0}: Error finding container 9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47: Status 404 returned error can't find the container with id 9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47
Oct 31 19:18:56.322636 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.322604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:18:56.326749 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.326724    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48cdb47_67b2_47ec_9433_d831d8393f53.slice/crio-37e2ed16b832ee0734f467e7533edfc276023ac659dae14cc9a98408773dabb7 WatchSource:0}: Error finding container 37e2ed16b832ee0734f467e7533edfc276023ac659dae14cc9a98408773dabb7: Status 404 returned error can't find the container with id 37e2ed16b832ee0734f467e7533edfc276023ac659dae14cc9a98408773dabb7
Oct 31 19:18:56.332531 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.332509    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9647c81_f8aa_470d_a5c7_02897a93fa31.slice/crio-877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11 WatchSource:0}: Error finding container 877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11: Status 404 returned error can't find the container with id 877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11
Oct 31 19:18:56.333411 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.333387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:18:56.335000 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.334971    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92bf693_c264_4e52_84a5_045804009257.slice/crio-e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e WatchSource:0}: Error finding container e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e: Status 404 returned error can't find the container with id e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e
Oct 31 19:18:56.342372 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.342245    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:18:56.344975 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.344929    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda237adff_6089_4d22_993f_95a9869a9771.slice/crio-6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965 WatchSource:0}: Error finding container 6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965: Status 404 returned error can't find the container with id 6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965
Oct 31 19:18:56.355087 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:18:56.355062    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a291037_a7e5_4cb8_bfde_a5aec293cbeb.slice/crio-b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312 WatchSource:0}: Error finding container b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312: Status 404 returned error can't find the container with id b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312
Oct 31 19:18:56.364017 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.363992    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:56.565882 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.565764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:56.566093 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.565953    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:56.566093 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.566024    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:57.566007675 +0000 UTC m=+3.959846331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:56.666843 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.666778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:56.667081 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.667032    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:56.667157 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.667133    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:57.667108628 +0000 UTC m=+4.060947295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:56.767384 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:56.767345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:56.767626 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.767594    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:56.767626 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.767626    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:56.767834 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.767644    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:56.767834 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:56.767709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:18:57.767688162 +0000 UTC m=+4.161526823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:57.071408 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.071336    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:55 +0000 UTC" deadline="2028-04-27 09:53:58.778718608 +0000 UTC"
Oct 31 19:18:57.071408 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.071406    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13046h35m1.707317136s"
Oct 31 19:18:57.157021 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.156931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:57.157212 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.157074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:18:57.157697 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.157674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:57.157827 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.157806    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:18:57.176977 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.176937    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7w6mv" event={"ID":"d48cdb47-67b2-47ec-9433-d831d8393f53","Type":"ContainerStarted","Data":"37e2ed16b832ee0734f467e7533edfc276023ac659dae14cc9a98408773dabb7"}
Oct 31 19:18:57.185855 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.185784    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6qz8h" event={"ID":"a237adff-6089-4d22-993f-95a9869a9771","Type":"ContainerStarted","Data":"6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965"}
Oct 31 19:18:57.195095 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.195044    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wm8m" event={"ID":"0312583f-6efa-4d8f-8921-d9deb835d69f","Type":"ContainerStarted","Data":"9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47"}
Oct 31 19:18:57.202170 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.202091    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerStarted","Data":"92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c"}
Oct 31 19:18:57.205936 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.205842    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2" event={"ID":"58e5ab86-50ff-419a-a4da-2c19408eb9ae","Type":"ContainerStarted","Data":"52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9"}
Oct 31 19:18:57.208698 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.208575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jv8rs" event={"ID":"22143d8e-6fcb-4563-8989-a29ddefbc053","Type":"ContainerStarted","Data":"fe55f2ab79aa2269dbb181052559e7229f18c97afb5d8323aba8303b4c639c36"}
Oct 31 19:18:57.215667 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.215566    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgtp6" event={"ID":"9a291037-a7e5-4cb8-bfde-a5aec293cbeb","Type":"ContainerStarted","Data":"b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312"}
Oct 31 19:18:57.218024 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.217943    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w6g8x" event={"ID":"d92bf693-c264-4e52-84a5-045804009257","Type":"ContainerStarted","Data":"e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e"}
Oct 31 19:18:57.227533 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.227499    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11"}
Oct 31 19:18:57.574582 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.574532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:57.574745 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.574708    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:57.574812 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.574772    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:59.574754262 +0000 UTC m=+5.968592922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:57.676100 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.675406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:57.676100 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.675628    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:57.676100 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.675705    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:59.675684243 +0000 UTC m=+6.069522907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:57.777068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:57.776794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:57.777507 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.777484    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:57.777591 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.777519    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:57.777591 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.777534    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:57.777732 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:57.777607    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:18:59.777587483 +0000 UTC m=+6.171426153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:58.159186 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:58.159131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:58.159711 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:58.159268    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:18:58.260438 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:58.260402    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"a1d262f6c678a43aaab0402c233dbed3","Type":"ContainerStarted","Data":"6499f0dbcb841ddbe104577a3c0f315147d57bf9a7e882a90ee2c0b18e5a40da"}
Oct 31 19:18:58.277882 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:58.277600    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal" podStartSLOduration=3.277579904 podStartE2EDuration="3.277579904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:58.277275596 +0000 UTC m=+4.671114276" watchObservedRunningTime="2026-10-31 19:18:58.277579904 +0000 UTC m=+4.671418577"
Oct 31 19:18:59.157972 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.157156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:59.157972 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.157310    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:18:59.157972 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.157796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:59.157972 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.157913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:18:59.260360 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.259636    2577 generic.go:356] "Generic (PLEG): container finished" podID="b12c7c191df1c0e91dfcc512de902352" containerID="c34a89b014166d1551dd958e877830bd73529594ef4a53692a3847879f7e3730" exitCode=0
Oct 31 19:18:59.260360 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.260213    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"b12c7c191df1c0e91dfcc512de902352","Type":"ContainerDied","Data":"c34a89b014166d1551dd958e877830bd73529594ef4a53692a3847879f7e3730"}
Oct 31 19:18:59.595605 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.594989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:18:59.595605 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.595165    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:59.595605 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.595237    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:03.595216716 +0000 UTC m=+9.989055387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:59.695794 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.695695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:18:59.695967 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.695860    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:59.695967 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.695931    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:03.695911332 +0000 UTC m=+10.089750009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:59.796705 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:18:59.796664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:18:59.796936 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.796916    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:59.797009 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.796949    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:59.797009 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.796963    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:59.797083 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:18:59.797036    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:03.797015903 +0000 UTC m=+10.190854580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:00.165910 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:00.165877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:00.166111 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:00.166027    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:00.272104 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:00.272019    2577 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="b5cdd9bbab1f147989757d0bcf7081c21000f34655845368c5b88fa878f77a65" exitCode=0
Oct 31 19:19:00.272598 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:00.272121    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerDied","Data":"b5cdd9bbab1f147989757d0bcf7081c21000f34655845368c5b88fa878f77a65"}
Oct 31 19:19:00.283407 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:00.282842    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"b12c7c191df1c0e91dfcc512de902352","Type":"ContainerStarted","Data":"ea8716b12c22b63567e5af2431200dda43bbb944abc4e469762ab6663cad8419"}
Oct 31 19:19:01.156919 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:01.156883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:01.157129 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:01.157040    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:01.157442 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:01.156822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:01.157576 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:01.157553    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:01.295374 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:01.295341    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgtp6" event={"ID":"9a291037-a7e5-4cb8-bfde-a5aec293cbeb","Type":"ContainerStarted","Data":"247c4a97841f928f3d0185bb48230348393c05e677d13442c5c68f997b1d77d0"}
Oct 31 19:19:01.312433 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:01.312372    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" podStartSLOduration=6.312352923 podStartE2EDuration="6.312352923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:00.318438686 +0000 UTC m=+6.712277368" watchObservedRunningTime="2026-10-31 19:19:01.312352923 +0000 UTC m=+7.706191599"
Oct 31 19:19:02.156377 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:02.156345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:02.156525 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:02.156482    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:02.302854 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:02.302770    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6qz8h" event={"ID":"a237adff-6089-4d22-993f-95a9869a9771","Type":"ContainerStarted","Data":"72fc07977248c08a4fef9493acb67ccb84bc5fc63856f671d47bdd365b3ab026"}
Oct 31 19:19:02.319262 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:02.319200    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vgtp6" podStartSLOduration=3.149018746 podStartE2EDuration="7.319176656s" totalImagesPullingTime="4.17015791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:01.313318069 +0000 UTC m=+7.707156746" watchObservedRunningTime="2026-10-31 19:19:02.319176656 +0000 UTC m=+8.713015334"
Oct 31 19:19:03.157744 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:03.157009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:03.157744 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.157153    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:03.157744 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:03.157576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:03.157744 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.157677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:03.312291 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:03.312255    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2" event={"ID":"58e5ab86-50ff-419a-a4da-2c19408eb9ae","Type":"ContainerStarted","Data":"5d9447026fd6e72a87cc3a0e26cbe838a4fdb1d1f8978f8951285cab7359a02f"}
Oct 31 19:19:03.629937 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:03.629899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:03.630104 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.630044    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:03.630150 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.630110    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:11.630091981 +0000 UTC m=+18.023930639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:03.730452 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:03.730415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:03.730629 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.730524    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:03.730629 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.730587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:11.730575226 +0000 UTC m=+18.124413893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:03.831308 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:03.831265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:03.831497 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.831438    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:19:03.831497 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.831459    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:19:03.831497 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.831473    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:03.831692 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:03.831536    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:11.831517547 +0000 UTC m=+18.225356225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:04.129886 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:04.129852    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:19:04.131196 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:04.131174    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:19:04.149869 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:04.149824    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6qz8h" podStartSLOduration=4.559631893 podStartE2EDuration="10.149810909s" totalImagesPullingTime="5.590179016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:02.320080205 +0000 UTC m=+8.713918881" watchObservedRunningTime="2026-10-31 19:19:04.149810909 +0000 UTC m=+10.543649584"
Oct 31 19:19:04.156872 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:04.156850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:04.157034 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:04.156941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:04.441728 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:04.441394    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 19:19:05.149147 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.148794    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:19:04.441413777Z","UUID":"6fee2c24-af24-4ecb-a156-cb577339c9e2","Handler":null,"Name":"","Endpoint":""}
Oct 31 19:19:05.150570 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.150521    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 19:19:05.150708 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.150581    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 19:19:05.156313 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.156288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:05.156442 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.156326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:05.156489 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:05.156442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:05.156621 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:05.156593    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:05.321159 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.321084    2577 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="65298ab3d58a1a6b8d98bdf46452e8354e9931802580fa2d701f3abc9b968bda" exitCode=0
Oct 31 19:19:05.321303 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.321203    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerDied","Data":"65298ab3d58a1a6b8d98bdf46452e8354e9931802580fa2d701f3abc9b968bda"}
Oct 31 19:19:05.326488 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.326463    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2" event={"ID":"58e5ab86-50ff-419a-a4da-2c19408eb9ae","Type":"ContainerStarted","Data":"b023d5767b8e12d1d84d94f27ff5060cf271d74c47cfedd00d90707faf433c7b"}
Oct 31 19:19:05.329088 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.328716    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jv8rs" event={"ID":"22143d8e-6fcb-4563-8989-a29ddefbc053","Type":"ContainerStarted","Data":"c03a698caa265e69e12eb6ea7e7bce55d08a0836adef54cec9214d542c1a7b17"}
Oct 31 19:19:05.330532 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.330496    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w6g8x" event={"ID":"d92bf693-c264-4e52-84a5-045804009257","Type":"ContainerStarted","Data":"a013469f253a56ef1dbac8b9beab227be26d02c4492e9d345e06343261303ac9"}
Oct 31 19:19:05.363700 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:05.363633    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jv8rs" podStartSLOduration=3.2152020869999998 podStartE2EDuration="11.363610372s" totalImagesPullingTime="8.148408285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:05.363527793 +0000 UTC m=+11.757366468" watchObservedRunningTime="2026-10-31 19:19:05.363610372 +0000 UTC m=+11.757449070"
Oct 31 19:19:06.156577 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.156524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:06.157101 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:06.156677    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:06.337078 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.336985    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2" event={"ID":"58e5ab86-50ff-419a-a4da-2c19408eb9ae","Type":"ContainerStarted","Data":"66364081f07c99b1af16c87192bf6555b8f5d9ac1975b926ae7ac08d8faea93f"}
Oct 31 19:19:06.340090 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.340061    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wm8m" event={"ID":"0312583f-6efa-4d8f-8921-d9deb835d69f","Type":"ContainerStarted","Data":"93e48b62376e2cc3cf5e5b32ce62f3e2c43242e3ae987b707e9cff96d8120e67"}
Oct 31 19:19:06.343530 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.343503    2577 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="5cd76d59b7e8552920586fffa957cf19d8e00e752558dc007a5999e60feda937" exitCode=0
Oct 31 19:19:06.343649 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.343623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerDied","Data":"5cd76d59b7e8552920586fffa957cf19d8e00e752558dc007a5999e60feda937"}
Oct 31 19:19:06.356702 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.356652    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-w6g8x" podStartSLOduration=4.090906666 podStartE2EDuration="12.356635918s" totalImagesPullingTime="8.265729252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:05.380438109 +0000 UTC m=+11.774276785" watchObservedRunningTime="2026-10-31 19:19:06.356635918 +0000 UTC m=+12.750474597"
Oct 31 19:19:06.356960 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.356930    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2" podStartSLOduration=3.5083980869999998 podStartE2EDuration="12.356921863s" totalImagesPullingTime="8.848523776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:06.356779552 +0000 UTC m=+12.750618219" watchObservedRunningTime="2026-10-31 19:19:06.356921863 +0000 UTC m=+12.750760533"
Oct 31 19:19:06.372659 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:06.372609    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9wm8m" podStartSLOduration=3.744410478 podStartE2EDuration="12.372595019s" totalImagesPullingTime="8.628184541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:06.372423643 +0000 UTC m=+12.766262322" watchObservedRunningTime="2026-10-31 19:19:06.372595019 +0000 UTC m=+12.766434117"
Oct 31 19:19:07.156998 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:07.156966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:07.157634 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:07.156966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:07.157634 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:07.157093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:07.157634 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:07.157147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:07.348399 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:07.348365    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7w6mv" event={"ID":"d48cdb47-67b2-47ec-9433-d831d8393f53","Type":"ContainerStarted","Data":"46bedb0ce84bc946458c887a48cfa0238ef462f95903f109cafa8992d466c6b2"}
Oct 31 19:19:07.368240 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:07.368172    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7w6mv" podStartSLOduration=2.706320947 podStartE2EDuration="13.368152518s" totalImagesPullingTime="10.661831571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:07.367559857 +0000 UTC m=+13.761398530" watchObservedRunningTime="2026-10-31 19:19:07.368152518 +0000 UTC m=+13.761991195"
Oct 31 19:19:08.156968 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:08.156754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:08.157130 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:08.157067    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:08.353342 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:08.353306    2577 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="4f6ecd3e7a2e1ad99fb384563bdbfaa37744a00d1470173bf2300fe56418ea13" exitCode=0
Oct 31 19:19:08.353529 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:08.353391    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerDied","Data":"4f6ecd3e7a2e1ad99fb384563bdbfaa37744a00d1470173bf2300fe56418ea13"}
Oct 31 19:19:09.156422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:09.156386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:09.156654 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:09.156589    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:09.156994 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:09.156971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:09.157122 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:09.157083    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:10.156390 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:10.156353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:10.156829 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:10.156490    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:11.156208 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.156175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:11.156363 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.156224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:11.156363 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.156333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:11.156475 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.156457    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:11.365761 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.365584    2577 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="71f86b18c9333db72682eafb2be1d80d4e0053ee32ed3f3a35e04c44c73a526c" exitCode=0
Oct 31 19:19:11.365871 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.365662    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerDied","Data":"71f86b18c9333db72682eafb2be1d80d4e0053ee32ed3f3a35e04c44c73a526c"}
Oct 31 19:19:11.368162 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.368135    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"0d49e27f6b940840042b25ab01b5ba89378e5164430cd8604f201a802def0c3e"}
Oct 31 19:19:11.368249 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.368202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"52be41702e8855a62ad3df7adc5412577d6421754b3e2eb774b24f389b0c5e6a"}
Oct 31 19:19:11.368249 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.368238    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"551772b1b8b5daced55ab74d45388ee4e1bf0f03ef45e0fc290882cb64924861"}
Oct 31 19:19:11.368334 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.368266    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"49075919b2e679a3bd1483f9f55e11dc9328343da11eac43b225c879211f53b5"}
Oct 31 19:19:11.690379 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.690340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:11.690535 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.690482    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:11.690607 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.690562    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.690527478 +0000 UTC m=+34.084366149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:11.791692 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.791655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:11.791862 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.791827    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:11.791922 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.791909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.791889325 +0000 UTC m=+34.185727999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:11.892824 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:11.892784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:11.892960 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.892944    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:19:11.893016 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.892965    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:19:11.893016 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.892975    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:11.893071 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:11.893025    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.893012426 +0000 UTC m=+34.286851094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:12.156290 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:12.156256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:12.156429 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:12.156391    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:12.372822 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:12.372787    2577 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="217c947e5ecf63095f13b67ee0ad52f898e3a15ada610bab35431cac80f73682" exitCode=0
Oct 31 19:19:12.373454 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:12.372860    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerDied","Data":"217c947e5ecf63095f13b67ee0ad52f898e3a15ada610bab35431cac80f73682"}
Oct 31 19:19:12.376139 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:12.376113    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"a801582b4a57491134f06712cf954149bd48d0bc8fcc2919817e15590a8f8044"}
Oct 31 19:19:12.376277 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:12.376202    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"124b871fd987175eaac39f4f1be80e595d478f00042bccc42e8acd5078c5d120"}
Oct 31 19:19:13.156165 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:13.156126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:13.156165 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:13.156152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:13.156343 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:13.156241    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:13.156399 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:13.156352    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:13.381756 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:13.381718    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qswht" event={"ID":"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf","Type":"ContainerStarted","Data":"b2d05a33571a4db22a9d956da4c4fc2a2535147cb7099fcea0a225f3fbe55ba4"}
Oct 31 19:19:13.406112 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:13.406067    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qswht" podStartSLOduration=8.444665739 podStartE2EDuration="19.406053875s" totalImagesPullingTime="10.961388136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:13.404935975 +0000 UTC m=+19.798774650" watchObservedRunningTime="2026-10-31 19:19:13.406053875 +0000 UTC m=+19.799892549"
Oct 31 19:19:14.157215 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:14.157182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:14.157384 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:14.157264    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:14.386708 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:14.386673    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"e7085ced060c260c6f4c015f48e4efea94f5fc232bbf1fb0d7e942e818ea597b"}
Oct 31 19:19:15.157073 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:15.157040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:15.157073 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:15.157077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:15.157325 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:15.157173    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:15.157325 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:15.157281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:16.156874 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:16.156836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:16.157283 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:16.156945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:16.394458 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:16.394420    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" event={"ID":"c9647c81-f8aa-470d-a5c7-02897a93fa31","Type":"ContainerStarted","Data":"5d55d85a44dd69979907b379cca9fb525fa3f036eb75a80c2e104aee6a607e89"}
Oct 31 19:19:16.394731 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:16.394709    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:19:16.394851 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:16.394782    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:19:16.411303 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:16.411209    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:19:16.427987 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:16.427935    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5" podStartSLOduration=7.6641256460000005 podStartE2EDuration="22.427922764s" totalImagesPullingTime="14.763797118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:16.427408556 +0000 UTC m=+22.821247232" watchObservedRunningTime="2026-10-31 19:19:16.427922764 +0000 UTC m=+22.821761438"
Oct 31 19:19:17.157110 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.157060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:17.157560 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.157071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:17.157560 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:17.157228    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:17.157560 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:17.157271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:17.371263 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.371231    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bz76x"]
Oct 31 19:19:17.371468 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.371352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:17.371468 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:17.371456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:17.374470 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.374442    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9xhcv"]
Oct 31 19:19:17.375136 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.375115    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bfhh2"]
Oct 31 19:19:17.395962 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.395938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:17.396095 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:17.396030    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:17.396095 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.395940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:17.396416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.396396    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:19:17.396487 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:17.396389    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:17.411690 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:17.411665    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:19:19.156592 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:19.156516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:19.157007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:19.156520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:19.157007 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:19.156681    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:19.157007 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:19.156722    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:19.157007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:19.156560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:19.157007 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:19.156816    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:21.156246 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:21.156202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:21.156674 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:21.156316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:21.156674 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:21.156333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:21.156674 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:21.156406    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:21.156674 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:21.156465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:21.156674 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:21.156539    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:23.156879 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:23.156698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:23.157327 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:23.156703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:23.157327 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:23.156966    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:23.157327 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:23.156729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:23.157327 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:23.157026    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:23.157327 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:23.157115    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:25.157083 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:25.157048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:25.157083 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:25.157081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:25.157629 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:25.157169    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:25.157629 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:25.157241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:25.157629 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:25.157348    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:25.157629 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:25.157456    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:25.486316 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:25.486218    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:19:25.486464 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:25.486367    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:19:25.486913 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:25.486890    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:19:27.156880 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:27.156838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:27.156880 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:27.156872    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:27.157432 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:27.156883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:27.157432 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.156951    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:27.157432 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.157009    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:27.157432 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.157082    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:27.704080 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:27.704042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:27.704240 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.704167    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:27.704240 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.704219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs podName:a989c545-d91d-4ca2-9dea-0637a462d204 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:59.704205822 +0000 UTC m=+66.098044476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs") pod "network-metrics-daemon-bfhh2" (UID: "a989c545-d91d-4ca2-9dea-0637a462d204") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:27.805414 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:27.805375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:27.805597 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.805529    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:27.805649 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.805605    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret podName:56c7eb65-3951-480e-88e2-1858cae684e6 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:59.805591745 +0000 UTC m=+66.199430398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret") pod "global-pull-secret-syncer-bz76x" (UID: "56c7eb65-3951-480e-88e2-1858cae684e6") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:27.906741 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:27.906696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:27.906927 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.906839    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:19:27.906927 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.906862    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:19:27.906927 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.906872    2577 projected.go:196] Error preparing data for projected volume kube-api-access-84d8m for pod openshift-network-diagnostics/network-check-target-9xhcv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:27.906927 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:27.906923    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m podName:2825d9bc-c104-43b0-bbe5-acbfd127f71c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:59.906909941 +0000 UTC m=+66.300748608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-84d8m" (UniqueName: "kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m") pod "network-check-target-9xhcv" (UID: "2825d9bc-c104-43b0-bbe5-acbfd127f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:28.424685 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:28.424658    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w6g8x_d92bf693-c264-4e52-84a5-045804009257/dns-node-resolver/0.log"
Oct 31 19:19:29.156926 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:29.156890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:29.157108 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:29.156890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:29.157108 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:29.157006    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:29.157108 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:29.156890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:29.157247 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:29.157060    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:29.157247 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:29.157154    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:29.193965 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:29.193940    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6qz8h_a237adff-6089-4d22-993f-95a9869a9771/node-ca/0.log"
Oct 31 19:19:29.653444 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:29.653411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgtp6_9a291037-a7e5-4cb8-bfde-a5aec293cbeb/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:29.819476 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:29.819439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgtp6_9a291037-a7e5-4cb8-bfde-a5aec293cbeb/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:30.221007 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:30.220966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgtp6_9a291037-a7e5-4cb8-bfde-a5aec293cbeb/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:30.341295 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:30.341261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgtp6_9a291037-a7e5-4cb8-bfde-a5aec293cbeb/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:30.445613 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:30.445579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgtp6_9a291037-a7e5-4cb8-bfde-a5aec293cbeb/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:31.156428 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:31.156396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:31.156878 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:31.156401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:31.156878 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:31.156500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:31.156878 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:31.156407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:31.156878 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:31.156629    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:31.156878 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:31.156713    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:33.156381 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:33.156353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:33.156381 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:33.156378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:33.156800 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:33.156389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:33.156800 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:33.156453    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:33.156800 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:33.156605    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:33.156800 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:33.156695    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:35.157057 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:35.157016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:35.157057 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:35.157039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:35.157619 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:35.157021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:35.157619 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:35.157139    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:35.157619 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:35.157208    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:35.157619 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:35.157279    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:37.156139 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:37.156100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:37.156568 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:37.156100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:37.156568 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:37.156226    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:37.156568 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:37.156100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:37.156568 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:37.156389    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:37.156568 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:37.156283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:39.156254 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:39.156216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:39.156694 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:39.156322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:39.156694 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:39.156337    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:39.156694 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:39.156349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:39.156694 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:39.156415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:39.156694 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:39.156503    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:41.156700 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:41.156668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:41.156700 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:41.156701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:41.157104 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:41.156784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:41.157104 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:41.156834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:41.157104 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:41.156914    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:41.157104 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:41.156959    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:43.156638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:43.156595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:43.157063 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:43.156593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:43.157063 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:43.156713    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xhcv" podUID="2825d9bc-c104-43b0-bbe5-acbfd127f71c"
Oct 31 19:19:43.157063 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:43.156611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:43.157214 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:43.157155    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz76x" podUID="56c7eb65-3951-480e-88e2-1858cae684e6"
Oct 31 19:19:43.157307 ip-10-0-138-35 kubenswrapper[2577]: E1031 19:19:43.157275    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfhh2" podUID="a989c545-d91d-4ca2-9dea-0637a462d204"
Oct 31 19:19:44.908866 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.908839    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeReady"
Oct 31 19:19:44.909286 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.908971    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 19:19:44.958366 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.958334    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vxmdl"]
Oct 31 19:19:44.961176 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.961148    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nx6db"]
Oct 31 19:19:44.961314 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.961186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:44.963882 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.963864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:44.966254 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.966227    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7xwb\""
Oct 31 19:19:44.966369 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.966301    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 19:19:44.966726 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.966713    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 19:19:44.972932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.972909    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vxmdl"]
Oct 31 19:19:44.978982 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.978296    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nx6db"]
Oct 31 19:19:44.985525 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.985501    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rrq6l"]
Oct 31 19:19:44.988164 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.988143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:44.997645 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:44.997622    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrq6l"]
Oct 31 19:19:45.066595 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.066493    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 19:19:45.066746 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.066628    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrxc5\""
Oct 31 19:19:45.066746 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.066632    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxmqk\""
Oct 31 19:19:45.066746 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.066662    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 19:19:45.066908 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.066892    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 19:19:45.067064 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.067047    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 19:19:45.067064 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.067062    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 19:19:45.067160 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.067066    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 19:19:45.067160 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.067140    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 19:19:45.127347 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-config-volume\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.127463 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c31f22-7c5d-43bb-b554-ad463e4702c5-data-volume\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.127463 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31c31f22-7c5d-43bb-b554-ad463e4702c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.127563 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc9qj\" (UniqueName: \"kubernetes.io/projected/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-api-access-gc9qj\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.127563 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnnf\" (UniqueName: \"kubernetes.io/projected/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-kube-api-access-drnnf\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.127641 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31c31f22-7c5d-43bb-b554-ad463e4702c5-crio-socket\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.127682 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-tmp-dir\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.127682 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5njb9\" (UniqueName: \"kubernetes.io/projected/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-kube-api-access-5njb9\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.127682 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-cert\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.127818 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.127818 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.127788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-metrics-tls\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.156631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.156603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:45.156631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.156623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:45.156711 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.156616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:45.166754 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166728    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:19:45.166754 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166744    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\""
Oct 31 19:19:45.166754 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166728    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\""
Oct 31 19:19:45.166940 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166815    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:19:45.166940 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166865    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\""
Oct 31 19:19:45.166940 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166885    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:19:45.166940 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.166939    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:19:45.228685 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31c31f22-7c5d-43bb-b554-ad463e4702c5-crio-socket\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.228685 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-tmp-dir\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.228880 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5njb9\" (UniqueName: \"kubernetes.io/projected/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-kube-api-access-5njb9\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.228880 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-cert\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.228958 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31c31f22-7c5d-43bb-b554-ad463e4702c5-crio-socket\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.228958 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.228958 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-metrics-tls\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.229093 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.228983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-config-volume\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.229093 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c31f22-7c5d-43bb-b554-ad463e4702c5-data-volume\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.229093 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-tmp-dir\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.229093 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31c31f22-7c5d-43bb-b554-ad463e4702c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.229283 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9qj\" (UniqueName: \"kubernetes.io/projected/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-api-access-gc9qj\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.229283 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnnf\" (UniqueName: \"kubernetes.io/projected/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-kube-api-access-drnnf\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.229412 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229389    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c31f22-7c5d-43bb-b554-ad463e4702c5-data-volume\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.229649 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229631    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.229827 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.229811    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-config-volume\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.232754 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.232731    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-metrics-tls\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.232965 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.232944    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31c31f22-7c5d-43bb-b554-ad463e4702c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.233005 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.232959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-cert\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.240125 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.240097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnnf\" (UniqueName: \"kubernetes.io/projected/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-kube-api-access-drnnf\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.240717 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.240699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9qj\" (UniqueName: \"kubernetes.io/projected/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-api-access-gc9qj\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.241094 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.241074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5njb9\" (UniqueName: \"kubernetes.io/projected/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-kube-api-access-5njb9\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.277434 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.277408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:45.285482 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.285464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:19:45.298889 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.298364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:19:45.567443 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.567406    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vxmdl"]
Oct 31 19:19:45.570488 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.570458    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nx6db"]
Oct 31 19:19:45.575754 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:45.575731    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c31f22_7c5d_43bb_b554_ad463e4702c5.slice/crio-98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18 WatchSource:0}: Error finding container 98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18: Status 404 returned error can't find the container with id 98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18
Oct 31 19:19:45.600185 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:45.600131    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrq6l"]
Oct 31 19:19:45.603129 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:45.603100    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a2e073_9cf5_4472_9ba8_df174e1bb3e0.slice/crio-7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23 WatchSource:0}: Error finding container 7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23: Status 404 returned error can't find the container with id 7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23
Oct 31 19:19:46.465057 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:46.465014    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrq6l" event={"ID":"39a2e073-9cf5-4472-9ba8-df174e1bb3e0","Type":"ContainerStarted","Data":"7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23"}
Oct 31 19:19:46.467569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:46.467296    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx6db" event={"ID":"31c31f22-7c5d-43bb-b554-ad463e4702c5","Type":"ContainerStarted","Data":"b4c2cde26eddfe4593ffc87bf6c2e977cf83891368ef6b233e215c44f3b4385c"}
Oct 31 19:19:46.467569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:46.467363    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx6db" event={"ID":"31c31f22-7c5d-43bb-b554-ad463e4702c5","Type":"ContainerStarted","Data":"98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18"}
Oct 31 19:19:46.469329 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:46.469300    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxmdl" event={"ID":"d2d2e4e2-5258-4be1-929f-1d92b689c0ff","Type":"ContainerStarted","Data":"471b77066069c7689da92646cd7e25e50d1c34cae160e4d202b80196a140d6b2"}
Oct 31 19:19:47.221519 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.221474    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"]
Oct 31 19:19:47.224839 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.224811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.230430 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.230401    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 19:19:47.230679 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.230658    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 19:19:47.230764 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.230722    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 19:19:47.231023 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.231001    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 19:19:47.231176 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.231158    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 19:19:47.231284 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.231211    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5xjgc\""
Oct 31 19:19:47.234693 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.234670    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"]
Oct 31 19:19:47.343241 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.343203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n66g\" (UniqueName: \"kubernetes.io/projected/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-kube-api-access-5n66g\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.343422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.343283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.343422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.343329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.343422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.343359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.444057 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.444010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.444309 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.444077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.444309 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.444099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.444309 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.444162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n66g\" (UniqueName: \"kubernetes.io/projected/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-kube-api-access-5n66g\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.445578 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.445533    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.447496 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.447476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.448239 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.448217    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.467449 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.467421    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n66g\" (UniqueName: \"kubernetes.io/projected/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-kube-api-access-5n66g\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.475112 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.475020    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx6db" event={"ID":"31c31f22-7c5d-43bb-b554-ad463e4702c5","Type":"ContainerStarted","Data":"aeb959a240b0e291e9204b86950e14824ae9172e8eb34b7e59d4fac8df832118"}
Oct 31 19:19:47.543369 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.543331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:19:47.840039 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:47.840000    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"]
Oct 31 19:19:47.844577 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:47.844505    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e36838_e7e0_4a4d_8aa5_7324bb6fcbe2.slice/crio-08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649 WatchSource:0}: Error finding container 08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649: Status 404 returned error can't find the container with id 08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649
Oct 31 19:19:48.479365 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.479334    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn" event={"ID":"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2","Type":"ContainerStarted","Data":"08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649"}
Oct 31 19:19:48.482083 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.482051    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx6db" event={"ID":"31c31f22-7c5d-43bb-b554-ad463e4702c5","Type":"ContainerStarted","Data":"6d3aed064ec6376b4d41749f32865617391146acba65825eb6495ab5805ee3d4"}
Oct 31 19:19:48.484144 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.484113    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxmdl" event={"ID":"d2d2e4e2-5258-4be1-929f-1d92b689c0ff","Type":"ContainerStarted","Data":"5e7b846329e978f859ba83435c5a138180232cf769668b1c7ebed4356029653e"}
Oct 31 19:19:48.484280 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.484170    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vxmdl" event={"ID":"d2d2e4e2-5258-4be1-929f-1d92b689c0ff","Type":"ContainerStarted","Data":"3320b540b8d5412713678a8d551c8814e6a9222d5063d72dd6b573f9c46bec58"}
Oct 31 19:19:48.484725 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.484704    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:48.523174 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.523112    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nx6db" podStartSLOduration=2.198870921 podStartE2EDuration="4.523097118s" totalImagesPullingTime="2.324226197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:19:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:48.502586285 +0000 UTC m=+54.896424960" watchObservedRunningTime="2026-10-31 19:19:48.523097118 +0000 UTC m=+54.916935785"
Oct 31 19:19:48.523408 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:48.523385    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vxmdl" podStartSLOduration=2.529716559 podStartE2EDuration="4.523379408s" totalImagesPullingTime="1.993662849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:48.52325137 +0000 UTC m=+54.917090045" watchObservedRunningTime="2026-10-31 19:19:48.523379408 +0000 UTC m=+54.917218077"
Oct 31 19:19:49.418360 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:49.418287    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:19:49.489993 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:49.489951    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrq6l" event={"ID":"39a2e073-9cf5-4472-9ba8-df174e1bb3e0","Type":"ContainerStarted","Data":"d6ed477ed1634b8dd0eca74f5487600e95484e5b9fde1382de6b54e02fbf08c8"}
Oct 31 19:19:49.492108 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:49.492078    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn" event={"ID":"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2","Type":"ContainerStarted","Data":"b787986adba8b9c8bf861266a4757dd7cd9b73e6991a1545d7e33d375eb1bd39"}
Oct 31 19:19:49.492246 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:49.492153    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn" event={"ID":"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2","Type":"ContainerStarted","Data":"0c0b826587921146f1c18fa5be161caf3d8493454ef96086c735fd81b05ce09d"}
Oct 31 19:19:49.507969 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:49.507900    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rrq6l" podStartSLOduration=2.6688729650000003 podStartE2EDuration="5.507877224s" totalImagesPullingTime="2.839004259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:49.50699567 +0000 UTC m=+55.900834349" watchObservedRunningTime="2026-10-31 19:19:49.507877224 +0000 UTC m=+55.901715899"
Oct 31 19:19:49.525557 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:49.525459    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn" podStartSLOduration=1.206839625 podStartE2EDuration="2.52543578s" totalImagesPullingTime="1.318596155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:49.525384194 +0000 UTC m=+55.919222876" watchObservedRunningTime="2026-10-31 19:19:49.52543578 +0000 UTC m=+55.919274479"
Oct 31 19:19:50.313873 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.313838    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7bffb8dc-997rs"]
Oct 31 19:19:50.316805 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.316785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.326508 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.326479    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 19:19:50.326651 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.326535    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 19:19:50.326651 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.326559    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 19:19:50.330046 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.329966    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dw4k4\""
Oct 31 19:19:50.330046 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.330011    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 19:19:50.330249 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.330229    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 19:19:50.330249 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.330230    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 19:19:50.330392 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.330375    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 19:19:50.331208 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.331190    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7bffb8dc-997rs"]
Oct 31 19:19:50.467898 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.467841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.467898 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.467902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.468120 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.468018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.468120 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.468070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.468120 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.468090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.468209 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.468162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.468209 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.468194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569109 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569109 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569109 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569109 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569728 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569728 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569728 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569728 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569666    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569917 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.569975 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.569893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.570090 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.570069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.571534 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.571515    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.571624 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.571523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.581995 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.581970    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.629745 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.629709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:19:50.903036 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:50.903002    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7bffb8dc-997rs"]
Oct 31 19:19:50.905965 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:50.905934    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1411a2_289b_443e_ba7c_663c8f931635.slice/crio-24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486 WatchSource:0}: Error finding container 24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486: Status 404 returned error can't find the container with id 24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486
Oct 31 19:19:51.500410 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.500363    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bffb8dc-997rs" event={"ID":"5d1411a2-289b-443e-ba7c-663c8f931635","Type":"ContainerStarted","Data":"24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486"}
Oct 31 19:19:51.608392 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.608359    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"]
Oct 31 19:19:51.611667 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.611641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.618771 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.618742    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 19:19:51.618944 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.618813    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rl85m\""
Oct 31 19:19:51.618944 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.618836    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 19:19:51.623155 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.623131    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"]
Oct 31 19:19:51.626219 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.626163    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"]
Oct 31 19:19:51.626329 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.626311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.638665 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.638643    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 19:19:51.645248 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.645213    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"]
Oct 31 19:19:51.646567 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.646518    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 19:19:51.646675 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.646664    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w7nnm\""
Oct 31 19:19:51.646745 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.646738    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 19:19:51.651993 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.651965    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l2kfm"]
Oct 31 19:19:51.655155 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.655137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.722703 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.722609    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 19:19:51.722703 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.722702    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 19:19:51.722948 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.722769    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mxbfw\""
Oct 31 19:19:51.739750 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.739711    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 19:19:51.777962 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.777824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.777962 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.777878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.778168 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-sys\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778168 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-wtmp\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778168 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-textfile\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778168 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-metrics-client-ca\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778363 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-tls\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778363 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvnd\" (UniqueName: \"kubernetes.io/projected/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-api-access-dkvnd\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.778429 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.778475 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.778475 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-root\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778564 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/162fd1bc-c40c-4001-8fa3-8258ed577b45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.778564 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.778564 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsq4c\" (UniqueName: \"kubernetes.io/projected/555bb1b5-4eea-4de4-b4a0-45613823a90b-kube-api-access-gsq4c\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.778564 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.778692 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/162fd1bc-c40c-4001-8fa3-8258ed577b45-kube-api-access-bww5n\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.778692 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.778692 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.778692 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.778645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879380 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-textfile\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-metrics-client-ca\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-tls\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvnd\" (UniqueName: \"kubernetes.io/projected/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-api-access-dkvnd\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.879569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.879569 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-root\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/162fd1bc-c40c-4001-8fa3-8258ed577b45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsq4c\" (UniqueName: \"kubernetes.io/projected/555bb1b5-4eea-4de4-b4a0-45613823a90b-kube-api-access-gsq4c\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/162fd1bc-c40c-4001-8fa3-8258ed577b45-kube-api-access-bww5n\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-textfile\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.879830 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.880422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-sys\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.880422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-sys\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.880422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.879937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-root\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.880422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.880120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-metrics-client-ca\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.880422 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.880223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.880631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.880604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-wtmp\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.881243 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.880668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.881243 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.880675    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/162fd1bc-c40c-4001-8fa3-8258ed577b45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.881243 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.880771    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-wtmp\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.881243 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.881113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.881243 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.881196    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.882416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.882397    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-tls\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.882792 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.882765    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.882900 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.882851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.883188 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.883168    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.883505 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.883474    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.883927 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.883905    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.894557 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.894521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/162fd1bc-c40c-4001-8fa3-8258ed577b45-kube-api-access-bww5n\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.895054 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.895008    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsq4c\" (UniqueName: \"kubernetes.io/projected/555bb1b5-4eea-4de4-b4a0-45613823a90b-kube-api-access-gsq4c\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:51.896617 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.896594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvnd\" (UniqueName: \"kubernetes.io/projected/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-api-access-dkvnd\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.932413 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.932383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:19:51.945019 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.944817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:19:51.973734 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:51.973697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:19:52.222356 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.222320    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"]
Oct 31 19:19:52.225629 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:52.225602    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162fd1bc_c40c_4001_8fa3_8258ed577b45.slice/crio-3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883 WatchSource:0}: Error finding container 3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883: Status 404 returned error can't find the container with id 3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883
Oct 31 19:19:52.231646 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.231620    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"]
Oct 31 19:19:52.235211 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:52.235183    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf300c7e3_5d90_4bc7_bd34_9d8e743d9f27.slice/crio-095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137 WatchSource:0}: Error finding container 095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137: Status 404 returned error can't find the container with id 095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137
Oct 31 19:19:52.506717 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.506621    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw" event={"ID":"162fd1bc-c40c-4001-8fa3-8258ed577b45","Type":"ContainerStarted","Data":"16f5e264da1761c07eeb3535690d18b34d451f2f3e4081ba7d1500ff789632e4"}
Oct 31 19:19:52.506717 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.506698    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw" event={"ID":"162fd1bc-c40c-4001-8fa3-8258ed577b45","Type":"ContainerStarted","Data":"c7ce8b3eddb797ac078c4e67548dcd92f9dc358e8ed7eee50a8cd1867daac0d2"}
Oct 31 19:19:52.506926 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.506726    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw" event={"ID":"162fd1bc-c40c-4001-8fa3-8258ed577b45","Type":"ContainerStarted","Data":"3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883"}
Oct 31 19:19:52.508334 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.508304    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kfm" event={"ID":"555bb1b5-4eea-4de4-b4a0-45613823a90b","Type":"ContainerStarted","Data":"02d0607a73c2931f3437fb89632fd90c5fe6c6848a5bf6c1e37a7f6dbb208fe8"}
Oct 31 19:19:52.509829 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.509804    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl" event={"ID":"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27","Type":"ContainerStarted","Data":"095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137"}
Oct 31 19:19:52.734076 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.734033    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:19:52.738049 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.738021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.743139 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743112    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 19:19:52.743416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743241    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nkp5g\""
Oct 31 19:19:52.743416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743312    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 19:19:52.743416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743333    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 19:19:52.743416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743244    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 19:19:52.743416 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743356    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 19:19:52.743849 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743828    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 19:19:52.743940 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743900    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 19:19:52.743940 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.743916    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 19:19:52.744727 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.744706    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 19:19:52.756937 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.756882    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:19:52.890755 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.890917 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.890917 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.890917 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.890917 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.890917 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.890979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.891003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.891028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.891057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.891171 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.891092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.919879 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.919847    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79fdb78f65-nscml"]
Oct 31 19:19:52.924449 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.924424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:52.936303 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.934527    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 19:19:52.936303 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.936227    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79fdb78f65-nscml"]
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.993626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.992829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:52.995910 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:52.995339    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.001834 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.000634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.001834 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.001221    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.001834 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.001790    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.006313 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.006257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.008626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.008440    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.010515 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.010491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.010626 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.010521    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.011126 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.011047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.011126 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.011048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.011612 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.011589    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.012030 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.012012    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.014745 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.014724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.069404 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.069360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:19:53.094068 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094212 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094212 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094212 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094368 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094368 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094368 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.094368 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.094345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.194859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.194890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.194935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.194951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.194977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.194996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.195037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.195055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.195789    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.196611    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.197509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.198276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.198932 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.198879    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.200022 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.199882    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.200022 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.199956    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.207904 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.207882    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.244881 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.244634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:19:53.392826 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.392752    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:19:53.397692 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:53.397649    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e67bf99_5b7c_4a4d_9b8d_097683e961fb.slice/crio-b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb WatchSource:0}: Error finding container b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb: Status 404 returned error can't find the container with id b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb
Oct 31 19:19:53.516991 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.516953    2577 generic.go:356] "Generic (PLEG): container finished" podID="555bb1b5-4eea-4de4-b4a0-45613823a90b" containerID="c6cc23202cf636085e4e2c7c80a3220cd776edfebe15e64ea2612c03af521f94" exitCode=0
Oct 31 19:19:53.517138 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.517051    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kfm" event={"ID":"555bb1b5-4eea-4de4-b4a0-45613823a90b","Type":"ContainerDied","Data":"c6cc23202cf636085e4e2c7c80a3220cd776edfebe15e64ea2612c03af521f94"}
Oct 31 19:19:53.519170 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.519139    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb"}
Oct 31 19:19:53.540944 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.540897    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79fdb78f65-nscml"]
Oct 31 19:19:53.549344 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:53.549233    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55adc602_d71a_4b6c_be3e_5033bd5d6393.slice/crio-feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6 WatchSource:0}: Error finding container feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6: Status 404 returned error can't find the container with id feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6
Oct 31 19:19:53.633111 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.633079    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b4c88497b-g2wmt"]
Oct 31 19:19:53.644475 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.644445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.648868    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.648999    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.649834    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.649910    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.649988    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ao4lshlj74aes\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.650028    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.650101    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dvsqv\""
Oct 31 19:19:53.650653 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.650575    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b4c88497b-g2wmt"]
Oct 31 19:19:53.800967 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.800930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.800989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-grpc-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4184cf94-99b2-4491-a90f-a523b8f9ba3c-metrics-client-ca\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cx4\" (UniqueName: \"kubernetes.io/projected/4184cf94-99b2-4491-a90f-a523b8f9ba3c-kube-api-access-88cx4\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.801299 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.801242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4184cf94-99b2-4491-a90f-a523b8f9ba3c-tmp\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902645 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902791 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cx4\" (UniqueName: \"kubernetes.io/projected/4184cf94-99b2-4491-a90f-a523b8f9ba3c-kube-api-access-88cx4\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902791 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4184cf94-99b2-4491-a90f-a523b8f9ba3c-tmp\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902791 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902791 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902971 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902971 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-grpc-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902971 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.902971 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.902948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4184cf94-99b2-4491-a90f-a523b8f9ba3c-metrics-client-ca\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.903225 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.903112    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4184cf94-99b2-4491-a90f-a523b8f9ba3c-tmp\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.903600 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.903574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4184cf94-99b2-4491-a90f-a523b8f9ba3c-metrics-client-ca\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.906026 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.905994    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.906125 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.906046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.906507 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.906468    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.906621 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.906566    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.906735 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.906686    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-grpc-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.907464 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.907439    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.914755 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.914732    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cx4\" (UniqueName: \"kubernetes.io/projected/4184cf94-99b2-4491-a90f-a523b8f9ba3c-kube-api-access-88cx4\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:53.988905 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.988873    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dvsqv\""
Oct 31 19:19:53.997268 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:53.997243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:54.300485 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.300453    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b4c88497b-g2wmt"]
Oct 31 19:19:54.305924 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:54.305891    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4184cf94_99b2_4491_a90f_a523b8f9ba3c.slice/crio-2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc WatchSource:0}: Error finding container 2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc: Status 404 returned error can't find the container with id 2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc
Oct 31 19:19:54.530826 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.530602    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw" event={"ID":"162fd1bc-c40c-4001-8fa3-8258ed577b45","Type":"ContainerStarted","Data":"a455f0fb7e75ca014609a777d90f5b51e3abe97cebbce080d71bf8d128bca9af"}
Oct 31 19:19:54.533679 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.533512    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc"}
Oct 31 19:19:54.536088 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.534983    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fdb78f65-nscml" event={"ID":"55adc602-d71a-4b6c-be3e-5033bd5d6393","Type":"ContainerStarted","Data":"feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6"}
Oct 31 19:19:54.539786 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.539122    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kfm" event={"ID":"555bb1b5-4eea-4de4-b4a0-45613823a90b","Type":"ContainerStarted","Data":"5d5ac39a1e86f5575a266a029c50d2361bb8123d1288ef6e77603ecb6125d88f"}
Oct 31 19:19:54.539786 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.539162    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l2kfm" event={"ID":"555bb1b5-4eea-4de4-b4a0-45613823a90b","Type":"ContainerStarted","Data":"71853123b6f68aca1ac3f60c9ff179d35d4f823239d7eaa513f71d5a79d34aa4"}
Oct 31 19:19:54.542282 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.541842    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bffb8dc-997rs" event={"ID":"5d1411a2-289b-443e-ba7c-663c8f931635","Type":"ContainerStarted","Data":"1c23d39da7f3fbd630002a27177e6227f12d22e31a31b8cc29fce678056c1e85"}
Oct 31 19:19:54.545363 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.545157    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl" event={"ID":"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27","Type":"ContainerStarted","Data":"e112a10ee1c6d145d6225e18bb0cb547d9cd9832fe3ab6f9a57732a5aeaf2f7f"}
Oct 31 19:19:54.545363 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.545201    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl" event={"ID":"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27","Type":"ContainerStarted","Data":"7eaa7718332a0501ed8918226eec7ee4062d1db681ed4f30fccc8358d9d79ef3"}
Oct 31 19:19:54.545363 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.545219    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl" event={"ID":"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27","Type":"ContainerStarted","Data":"a44011ae4d8c6f8be6142334b8d7910effe010565fb43897cc4d1a7a8eb316e9"}
Oct 31 19:19:54.554644 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.554586    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw" podStartSLOduration=2.156032832 podStartE2EDuration="3.55457136s" totalImagesPullingTime="1.398538528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:54.553849639 +0000 UTC m=+60.947688314" watchObservedRunningTime="2026-10-31 19:19:54.55457136 +0000 UTC m=+60.948410034"
Oct 31 19:19:54.601315 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.601253    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c7bffb8dc-997rs" podStartSLOduration=1.100684229 podStartE2EDuration="4.601232879s" totalImagesPullingTime="3.50054865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:54.5779964 +0000 UTC m=+60.971835076" watchObservedRunningTime="2026-10-31 19:19:54.601232879 +0000 UTC m=+60.995071555"
Oct 31 19:19:54.601617 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.601581    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l2kfm" podStartSLOduration=2.400505988 podStartE2EDuration="3.601568194s" totalImagesPullingTime="1.113360866s" totalInitContainerRuntime="87.70134ms" isStatefulPod=true podCreationTimestamp="2026-10-31 19:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:54.59970426 +0000 UTC m=+60.993542935" watchObservedRunningTime="2026-10-31 19:19:54.601568194 +0000 UTC m=+60.995406871"
Oct 31 19:19:54.619499 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:54.619390    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl" podStartSLOduration=1.783302384 podStartE2EDuration="3.619371381s" totalImagesPullingTime="1.836068997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:54.618741592 +0000 UTC m=+61.012580269" watchObservedRunningTime="2026-10-31 19:19:54.619371381 +0000 UTC m=+61.013210057"
Oct 31 19:19:55.550433 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:55.550398    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6" exitCode=0
Oct 31 19:19:55.550927 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:55.550484    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6"}
Oct 31 19:19:56.036462 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.036427    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-674ffd47d7-5p6sd"]
Oct 31 19:19:56.042016 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.041981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.055795 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.055771    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 19:19:56.055963 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.055950    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 19:19:56.057481 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.057457    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-674ffd47d7-5p6sd"]
Oct 31 19:19:56.079950 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.079919    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 19:19:56.080158 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.080095    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 19:19:56.080238 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.080185    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kpnvw\""
Oct 31 19:19:56.080281 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.080265    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5n2nlk2eihv8p\""
Oct 31 19:19:56.126888 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.126851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-metrics-server-audit-profiles\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.127027 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.126963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-tls\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.127027 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.127015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-client-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.127142 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.127064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.127181 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.127158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-client-certs\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.127314 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.127289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98bdd4d0-f6c1-4a72-a603-e4928359958d-audit-log\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.127425 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.127339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5c2g\" (UniqueName: \"kubernetes.io/projected/98bdd4d0-f6c1-4a72-a603-e4928359958d-kube-api-access-s5c2g\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228429 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-metrics-server-audit-profiles\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-tls\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-client-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-client-certs\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98bdd4d0-f6c1-4a72-a603-e4928359958d-audit-log\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.228631 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.228617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5c2g\" (UniqueName: \"kubernetes.io/projected/98bdd4d0-f6c1-4a72-a603-e4928359958d-kube-api-access-s5c2g\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.229929 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.229897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-metrics-server-audit-profiles\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.230861 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.230840    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98bdd4d0-f6c1-4a72-a603-e4928359958d-audit-log\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.231613 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.231587    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.233695 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.233671    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-tls\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.233941 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.233912    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-client-certs\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.234582 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.234557    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-client-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.240765 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.240736    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5c2g\" (UniqueName: \"kubernetes.io/projected/98bdd4d0-f6c1-4a72-a603-e4928359958d-kube-api-access-s5c2g\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.354519 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.354448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:19:56.557961 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.557819    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"be6a1f1078d4eb8f7faf3cf154aa00703dfe1959bef0653bdccc4720e92279ac"}
Oct 31 19:19:56.560174 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.560149    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fdb78f65-nscml" event={"ID":"55adc602-d71a-4b6c-be3e-5033bd5d6393","Type":"ContainerStarted","Data":"25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e"}
Oct 31 19:19:56.581046 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.580999    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79fdb78f65-nscml" podStartSLOduration=2.567752765 podStartE2EDuration="4.580986164s" totalImagesPullingTime="2.013233399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:56.579855291 +0000 UTC m=+62.973693965" watchObservedRunningTime="2026-10-31 19:19:56.580986164 +0000 UTC m=+62.974824838"
Oct 31 19:19:56.645476 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:56.645444    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-674ffd47d7-5p6sd"]
Oct 31 19:19:56.647734 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:56.647702    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bdd4d0_f6c1_4a72_a603_e4928359958d.slice/crio-acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d WatchSource:0}: Error finding container acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d: Status 404 returned error can't find the container with id acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d
Oct 31 19:19:57.571567 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.571400    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"533c541dc0dcd1e76022658b0d812ac2b36525c8e84b859ef2b802c4daa3afa3"}
Oct 31 19:19:57.571567 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.571499    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"7918c7692d64e8c7f48f1aca98d612ad09cb30787c6c279ee1bef17103089a34"}
Oct 31 19:19:57.574986 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.574959    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43"}
Oct 31 19:19:57.575108 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.575032    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c"}
Oct 31 19:19:57.575108 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.575073    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d"}
Oct 31 19:19:57.575201 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.575112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26"}
Oct 31 19:19:57.575201 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.575152    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a"}
Oct 31 19:19:57.576572 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.576526    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd" event={"ID":"98bdd4d0-f6c1-4a72-a603-e4928359958d","Type":"ContainerStarted","Data":"acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d"}
Oct 31 19:19:57.903055 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.902978    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:19:57.907892 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.907862    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917051    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917160    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917220    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917283    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917288    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917359    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917363    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dbvnt\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917384    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4405tprkifkut\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917391    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917433    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917455    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917438    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917497    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 19:19:57.917512 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.917497    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 19:19:57.920342 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.920317    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 19:19:57.920505 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.920486    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 19:19:57.927164 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.926997    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:19:57.953228 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953464 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953593 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953593 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953593 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953711 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953750 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953786 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953815 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953854 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953854 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953925 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953925 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.953986 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.954014 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.953991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.954045 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.954019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.954081 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.954051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.954111 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.954081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.954144 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.954108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:57.954144 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:57.954127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.056790 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.057823 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.057823 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.057823 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.055978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.057823 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.056823    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.057823 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.057084    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.057823 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.057751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.058141 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.058020    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.060762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.058838    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.060762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.058880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.060762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.059272    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.060762 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.059321    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.062458 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.062428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.063238 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.063213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.063358 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.063337    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.063434 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.063405    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.064332 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.064150    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.064332 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.064243    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.064970 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.064659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.064970 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.064927    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.065568 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.065512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.066052 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.065999    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.067217 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.067097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.068873 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.068842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.253851 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.253765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:19:58.545898 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.545866    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:19:58.560579 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:19:58.560527    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdcc1f0_1892_4f50_9b50_2c6b864b502e.slice/crio-b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba WatchSource:0}: Error finding container b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba: Status 404 returned error can't find the container with id b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba
Oct 31 19:19:58.583065 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.582909    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"c82a80c304e1cd6c7eeba649f2882183790c74a5cff7addc512c8d3f8138d71b"}
Oct 31 19:19:58.583065 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.582994    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"5c9557a464d1c5951d958dd29159ae346b9764343fbecef4ccdd6273a8661549"}
Oct 31 19:19:58.583065 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.583047    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:19:58.583415 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.583086    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" event={"ID":"4184cf94-99b2-4491-a90f-a523b8f9ba3c","Type":"ContainerStarted","Data":"0b664a12143b9ad19c827813da2967a5c11c514e600461861790394a47b86539"}
Oct 31 19:19:58.590974 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.590945    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerStarted","Data":"f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7"}
Oct 31 19:19:58.592164 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.592140    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba"}
Oct 31 19:19:58.610159 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:58.610107    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt" podStartSLOduration=2.507199847 podStartE2EDuration="5.610093721s" totalImagesPullingTime="3.102893874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:58.608800545 +0000 UTC m=+65.002639211" watchObservedRunningTime="2026-10-31 19:19:58.610093721 +0000 UTC m=+65.003932395"
Oct 31 19:19:59.497444 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.497414    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:19:59.517410 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.517359    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.944661808 podStartE2EDuration="7.51734222s" totalImagesPullingTime="3.556338844s" totalInitContainerRuntime="16.341568ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:58.649524044 +0000 UTC m=+65.043362719" watchObservedRunningTime="2026-10-31 19:19:59.51734222 +0000 UTC m=+65.911180915"
Oct 31 19:19:59.597506 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.597473    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd" event={"ID":"98bdd4d0-f6c1-4a72-a603-e4928359958d","Type":"ContainerStarted","Data":"72a2249d17a0ef0a0867a86419b0ac7156cad7c69d635719a51ce47df800780c"}
Oct 31 19:19:59.599002 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.598974    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e" exitCode=0
Oct 31 19:19:59.599106 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.599049    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"}
Oct 31 19:19:59.617191 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.617141    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd" podStartSLOduration=1.700258862 podStartE2EDuration="3.617126779s" totalImagesPullingTime="1.916867917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:59.615851435 +0000 UTC m=+66.009690120" watchObservedRunningTime="2026-10-31 19:19:59.617126779 +0000 UTC m=+66.010965454"
Oct 31 19:19:59.776835 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.776749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:59.779319 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.779300    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:19:59.789590 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.789534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:59.877228 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.877200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:59.878170 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.878152    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\""
Oct 31 19:19:59.879278 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.879262    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:19:59.886531 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.886508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:19:59.889825 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.889804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:19:59.978386 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.978352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:19:59.980613 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.980597    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:19:59.990590 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:19:59.990561    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:20:00.008365 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.008335    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:00.161374 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:20:00.161344    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda989c545_d91d_4ca2_9dea_0637a462d204.slice/crio-70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27 WatchSource:0}: Error finding container 70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27: Status 404 returned error can't find the container with id 70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27
Oct 31 19:20:00.162157 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.162134    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bfhh2"]
Oct 31 19:20:00.170847 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.170819    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\""
Oct 31 19:20:00.178906 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.178883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:00.181316 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.181256    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\""
Oct 31 19:20:00.190040 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.189887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:00.458317 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.458293    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9xhcv"]
Oct 31 19:20:00.460792 ip-10-0-138-35 kubenswrapper[2577]: W1031 19:20:00.460757    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2825d9bc_c104_43b0_bbe5_acbfd127f71c.slice/crio-e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b WatchSource:0}: Error finding container e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b: Status 404 returned error can't find the container with id e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b
Oct 31 19:20:00.482994 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.482947    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bz76x"]
Oct 31 19:20:00.606773 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.606734    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9xhcv" event={"ID":"2825d9bc-c104-43b0-bbe5-acbfd127f71c","Type":"ContainerStarted","Data":"e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b"}
Oct 31 19:20:00.608239 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.608206    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bfhh2" event={"ID":"a989c545-d91d-4ca2-9dea-0637a462d204","Type":"ContainerStarted","Data":"70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27"}
Oct 31 19:20:00.610649 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.610615    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bz76x" event={"ID":"56c7eb65-3951-480e-88e2-1858cae684e6","Type":"ContainerStarted","Data":"7d9d3a4c5bcca5179e83dddf94b9dc5b99865faea2d117799619db94dd63392f"}
Oct 31 19:20:00.629943 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.629906    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:00.630145 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.629977    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:00.636126 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:00.636093    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:01.621319 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:01.619946    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bfhh2" event={"ID":"a989c545-d91d-4ca2-9dea-0637a462d204","Type":"ContainerStarted","Data":"fabc72f9db155099e3443aef0a180d24f57a6db1ec63c3767fb74883510bd6b6"}
Oct 31 19:20:01.621319 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:01.620096    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bfhh2" event={"ID":"a989c545-d91d-4ca2-9dea-0637a462d204","Type":"ContainerStarted","Data":"8ceafff27ee576700493dd6f7f190605fbd8253130d72648ef5ad0c40164f44d"}
Oct 31 19:20:01.625890 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:01.625868    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:01.663182 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:01.663128    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bfhh2" podStartSLOduration=66.6542337 podStartE2EDuration="1m7.663105715s" totalImagesPullingTime="1.008872015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:01.640884073 +0000 UTC m=+68.034722747" watchObservedRunningTime="2026-10-31 19:20:01.663105715 +0000 UTC m=+68.056944397"
Oct 31 19:20:03.245974 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:03.245933    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:03.246455 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:03.246002    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:03.251881 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:03.251854    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:03.639724 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:03.639698    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:03.710193 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:03.710157    2577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7bffb8dc-997rs"]
Oct 31 19:20:04.606624 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.606525    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:04.644638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.644204    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"}
Oct 31 19:20:04.644638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.644290    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"}
Oct 31 19:20:04.644638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.644330    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"}
Oct 31 19:20:04.644638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.644389    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"}
Oct 31 19:20:04.644638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.644428    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"}
Oct 31 19:20:04.644638 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.644481    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerStarted","Data":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"}
Oct 31 19:20:04.686359 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:04.686260    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.485746996 podStartE2EDuration="7.686237789s" totalImagesPullingTime="4.19274353s" totalInitContainerRuntime="7.747263ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:04.683203817 +0000 UTC m=+71.077042492" watchObservedRunningTime="2026-10-31 19:20:04.686237789 +0000 UTC m=+71.080076464"
Oct 31 19:20:05.653492 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:05.653449    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9xhcv" event={"ID":"2825d9bc-c104-43b0-bbe5-acbfd127f71c","Type":"ContainerStarted","Data":"936ca776a50fd8e10ce37506bdcb35e35b42a0105488cdf614f03cea732bad63"}
Oct 31 19:20:05.653996 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:05.653580    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:05.671821 ip-10-0-138-35 kubenswrapper[2577]: I1031 19:20:05.671757    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9xhcv" podStartSLOduration=67.520881224 podStartE2EDuration="1m11.67173868s" totalImagesPullingTime="4.150857456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:05.671224412 +0000 UTC m=+72.065063085" watchObservedRunningTime="2026-10-31 19:20:05.67173868 +0000 UTC m=+72.065577356"
Oct 31 19:20:06.479791 ip-10-0-138-35 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 19:20:06.493145 ip-10-0-138-35 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 19:20:06.493318 ip-10-0-138-35 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 19:20:06.493369 ip-10-0-138-35 systemd[1]: kubelet.service: Consumed 4.721s CPU time.
Oct 31 19:20:06.505410 ip-10-0-138-35 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 19:20:06.569054 ip-10-0-138-35 kubenswrapper[5958]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:20:06.569054 ip-10-0-138-35 kubenswrapper[5958]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 19:20:06.569054 ip-10-0-138-35 kubenswrapper[5958]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:20:06.569054 ip-10-0-138-35 kubenswrapper[5958]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:20:06.572913 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572815    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.572913 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572908    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.572913 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572913    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.572913 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572919    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572926    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572930    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572935    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572939    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572943    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572947    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572951    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572955    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572959    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572963    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572967    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572971    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572975    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572979    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572983    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572987    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572991    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.572995    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573000    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.573199 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573004    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573008    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573013    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573017    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573021    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573025    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573029    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573033    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573046    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573051    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573055    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573059    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573065    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573071    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573077    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573082    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573087    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573091    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573096    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.574072 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573103    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573109    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573115    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573121    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573127    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573131    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573135    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573140    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573146    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573150    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573154    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573159    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573164    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573168    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573173    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573177    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573181    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573186    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573190    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.574959 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573195    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573199    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573203    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573207    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573211    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573216    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573220    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573224    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573229    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573233    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573238    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573243    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573247    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573252    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573256    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573260    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573265    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573269    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573273    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573277    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.575819 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573282    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573286    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573293    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573297    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573301    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573306    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573311    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573315    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573319    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573323    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573328    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573332    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573336    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573341    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573348    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573352    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573357    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573361    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573365    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.576739 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573370    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573375    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573379    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573383    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573390    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.573394    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574710    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574723    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574728    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574733    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574737    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574741    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574747    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574751    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574756    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574760    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574766    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574770    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574776    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574780    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.577616 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574784    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574789    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574794    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574798    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574803    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574808    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574812    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574821    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574825    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574830    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574834    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574838    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574843    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574847    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574852    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574856    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574860    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574865    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574869    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574873    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.578597 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574878    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574883    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574890    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574897    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574902    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574906    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574911    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574915    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574919    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574924    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574930    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574934    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574938    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574943    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574947    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574952    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574956    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574960    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574964    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.579454 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574970    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574974    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574978    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574982    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574986    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574991    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.574996    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575001    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575006    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575010    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575014    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575019    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575023    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575027    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575032    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575037    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575041    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575045    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575050    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.580027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575054    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575058    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575063    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575067    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575072    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575077    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575081    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575085    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575090    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575096    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575102    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575107    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575112    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575118    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575122    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575127    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575131    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575135    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575140    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.580496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575144    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575149    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575153    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575158    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575162    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575166    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575173    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575177    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575182    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575187    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575191    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575195    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575199    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575203    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.575208    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576050    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576059    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576067    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576071    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.580999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576076    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576081    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576085    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576090    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576094    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576098    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576103    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576108    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576112    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576117    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576121    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576125    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576130    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576134    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576138    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576142    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576146    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576151    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576156    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576160    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.581530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576164    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576171    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576176    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576180    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576184    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576188    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576193    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576197    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576201    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576206    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576210    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576224    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576228    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576232    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576237    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576241    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576246    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576251    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576255    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576259    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.582030 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576264    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576268    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576272    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576278    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576282    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576287    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576291    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576295    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576300    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576305    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576309    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576313    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576317    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576328    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576332    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576337    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576341    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576345    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576349    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.582622 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576355    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576360    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576364    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576369    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576373    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576386    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576390    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576395    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576399    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576404    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576408    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576412    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576416    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576421    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576425    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576429    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576433    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576438    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576444    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.583124 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576451    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576457    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576462    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576467    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576472    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576479    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576483    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576489    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576493    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576497    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576501    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576505    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576509    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576513    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576518    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576522    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576526    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576530    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576535    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.583620 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576555    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576560    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576564    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576569    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.576575    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576719    5958 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576736    5958 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576746    5958 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576753    5958 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576760    5958 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576765    5958 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576772    5958 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576778    5958 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576784    5958 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576789    5958 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576795    5958 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576800    5958 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576805    5958 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576810    5958 flags.go:64] FLAG: --cgroup-root=""
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576815    5958 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576821    5958 flags.go:64] FLAG: --client-ca-file=""
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576825    5958 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576830    5958 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576837    5958 flags.go:64] FLAG: --cluster-domain=""
Oct 31 19:20:06.584087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576841    5958 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576847    5958 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576852    5958 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576858    5958 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576864    5958 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576869    5958 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576874    5958 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576879    5958 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576884    5958 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576891    5958 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576895    5958 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576901    5958 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576908    5958 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576915    5958 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576920    5958 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576925    5958 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576930    5958 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576935    5958 flags.go:64] FLAG: --enable-server="true"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576939    5958 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576947    5958 flags.go:64] FLAG: --event-burst="100"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576952    5958 flags.go:64] FLAG: --event-qps="50"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576957    5958 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576962    5958 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576967    5958 flags.go:64] FLAG: --eviction-hard=""
Oct 31 19:20:06.584735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576974    5958 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576979    5958 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576984    5958 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576989    5958 flags.go:64] FLAG: --eviction-soft=""
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.576995    5958 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577000    5958 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577006    5958 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577011    5958 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577016    5958 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577020    5958 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577025    5958 flags.go:64] FLAG: --feature-gates=""
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577032    5958 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577037    5958 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577042    5958 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577048    5958 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577053    5958 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577058    5958 flags.go:64] FLAG: --help="false"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577063    5958 flags.go:64] FLAG: --hostname-override="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577070    5958 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577075    5958 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577082    5958 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577088    5958 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577094    5958 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 19:20:06.585325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577099    5958 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577104    5958 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577108    5958 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577113    5958 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577118    5958 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577123    5958 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577129    5958 flags.go:64] FLAG: --kube-reserved=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577134    5958 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577138    5958 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577144    5958 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577148    5958 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577153    5958 flags.go:64] FLAG: --lock-file=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577158    5958 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577163    5958 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577168    5958 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577183    5958 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577188    5958 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577194    5958 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577199    5958 flags.go:64] FLAG: --logging-format="text"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577204    5958 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577209    5958 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577214    5958 flags.go:64] FLAG: --manifest-url=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577219    5958 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577228    5958 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577233    5958 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 19:20:06.585893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577248    5958 flags.go:64] FLAG: --max-pods="110"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577253    5958 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577258    5958 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577264    5958 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577272    5958 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577277    5958 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577282    5958 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577287    5958 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577298    5958 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577303    5958 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577309    5958 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577314    5958 flags.go:64] FLAG: --pod-cidr=""
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577319    5958 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577324    5958 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577330    5958 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577335    5958 flags.go:64] FLAG: --port="10250"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577340    5958 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577345    5958 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01d9b77015ae7a388"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577351    5958 flags.go:64] FLAG: --qos-reserved=""
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577356    5958 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577361    5958 flags.go:64] FLAG: --register-node="true"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577366    5958 flags.go:64] FLAG: --register-with-taints=""
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577373    5958 flags.go:64] FLAG: --registry-burst="10"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577377    5958 flags.go:64] FLAG: --registry-qps="5"
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577382    5958 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577387    5958 flags.go:64] FLAG: --reserved-memory=""
Oct 31 19:20:06.586507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577419    5958 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577426    5958 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577432    5958 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577437    5958 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577442    5958 flags.go:64] FLAG: --runonce="false"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577448    5958 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577453    5958 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577458    5958 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577463    5958 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577468    5958 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577477    5958 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577486    5958 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577492    5958 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577496    5958 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577501    5958 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577506    5958 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577510    5958 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577516    5958 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577521    5958 flags.go:64] FLAG: --system-cgroups=""
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577526    5958 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577534    5958 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577539    5958 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577559    5958 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577565    5958 flags.go:64] FLAG: --tls-min-version=""
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577570    5958 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 19:20:06.587153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577575    5958 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577579    5958 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577585    5958 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577590    5958 flags.go:64] FLAG: --v="2"
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577597    5958 flags.go:64] FLAG: --version="false"
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577604    5958 flags.go:64] FLAG: --vmodule=""
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577610    5958 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.577616    5958 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577844    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577853    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577859    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577864    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577869    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577873    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577878    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577882    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577887    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577896    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577904    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577909    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577914    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.587779 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577918    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577923    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577927    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577932    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577936    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577940    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577944    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577948    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577953    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577958    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577962    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577966    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577971    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577977    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577983    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577988    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577992    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.577997    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578003    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.588295 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578007    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578011    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578016    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578020    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578024    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578028    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578033    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578037    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578042    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578048    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578055    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578067    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578071    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578075    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578080    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578084    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578089    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578093    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578097    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.588971 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578102    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578106    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578110    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578114    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578119    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578123    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578127    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578132    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578136    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578140    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578145    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578149    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578153    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578157    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578162    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578166    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578170    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578175    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578179    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578184    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.589652 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578188    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578192    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578198    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578204    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578209    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578214    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578219    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578223    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578227    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578231    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578238    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578245    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578249    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578255    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578261    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578266    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578272    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578286    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578291    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.590231 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578295    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578300    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578304    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578308    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578313    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578318    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578322    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578326    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578330    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578336    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578340    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578345    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578349    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578354    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578358    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578364    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.590807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.578413    5958 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578482    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578490    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578495    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578500    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578505    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578509    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578514    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578518    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578522    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578527    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578531    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578535    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578539    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578562    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578566    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578571    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578575    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578580    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578584    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.591321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578589    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578593    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578597    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578601    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578606    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578609    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578613    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578617    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578622    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578626    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578630    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578635    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578644    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578648    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578653    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578657    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578663    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578669    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578676    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.591967 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578681    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578685    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578690    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578695    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578699    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578703    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578708    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578712    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578716    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578721    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578725    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578730    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578734    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578739    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578743    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578748    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578753    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578758    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578762    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578766    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.592432 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578771    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578775    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578780    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578784    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578788    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578795    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578799    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578804    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578808    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578813    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578819    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578824    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578829    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578833    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578837    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578842    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578846    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578851    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578855    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578859    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.593001 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578864    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578868    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578872    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578877    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578881    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578885    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578890    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578894    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578898    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578905    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578911    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578916    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578920    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578925    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578929    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578934    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578938    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578948    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578952    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.593645 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578957    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578961    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578965    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578971    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578977    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578981    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578986    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578990    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.578994    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.583481    5958 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.583500    5958 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.583519    5958 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.583525    5958 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583607    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583612    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583615    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583619    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583622    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583625    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.594182 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583628    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583631    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583633    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583636    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583639    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583643    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583648    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583652    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583656    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583659    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583662    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583665    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583668    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583670    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583673    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583676    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583679    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583682    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583685    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.594718 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583687    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583690    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583693    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583695    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583704    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583707    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583709    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583712    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583715    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583717    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583720    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583722    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583725    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583727    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583730    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583733    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583736    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583738    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583741    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583744    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.595222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583747    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583749    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583752    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583755    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583757    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583760    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583764    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583767    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583770    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583773    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583776    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583779    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583781    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583784    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583787    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583789    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583792    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583800    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583803    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.595778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583805    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583808    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583810    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583813    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583816    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583818    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583821    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583823    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583826    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583829    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583831    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583834    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583837    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583840    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583843    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583845    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583848    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583851    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583853    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.596290 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583856    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583859    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583861    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583864    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583867    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583869    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583872    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583875    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583877    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583880    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583882    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583885    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583892    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583897    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583901    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583905    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583909    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583913    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583917    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583921    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.597014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583926    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583930    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.583934    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.583964    5958 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584164    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584170    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584173    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584176    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584179    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584181    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584184    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584187    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584190    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584193    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584196    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584198    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.597623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584201    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584203    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584206    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584209    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584211    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584214    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584216    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584219    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584221    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584230    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584233    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584236    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584239    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584241    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584244    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584248    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584252    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584255    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584258    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.598078 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584261    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584264    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584267    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584270    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584273    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584276    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584279    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584281    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584284    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584287    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584289    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584292    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584294    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584297    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584299    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584302    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584305    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584307    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584309    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584312    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.598576 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584315    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584317    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584320    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584329    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584331    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584335    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584338    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584340    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584343    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584345    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584348    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584350    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584353    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584355    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584358    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584360    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584363    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584365    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584368    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.599076 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584371    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584373    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584376    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584379    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584381    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584389    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584392    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584395    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584397    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584400    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584402    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584405    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584407    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584410    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584413    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584416    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584418    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584426    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584438    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.599624 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584441    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584444    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584447    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584449    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584452    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584455    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584457    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584460    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584462    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584465    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584467    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584470    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584472    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584477    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584480    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584483    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.600087 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584486    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.584510    5958 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584559    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584563    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584567    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584570    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584573    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584579    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584581    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584584    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584587    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584590    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584592    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584595    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584597    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584606    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:20:06.600555 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584609    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584611    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584614    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584617    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584619    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584622    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584625    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584628    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584630    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584633    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584636    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584639    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584641    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584644    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584646    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584649    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584652    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584654    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584656    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:20:06.600951 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584659    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584662    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584664    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584666    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584672    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584675    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584679    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584682    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584685    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584688    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584691    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584693    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584696    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584704    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584707    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584709    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584712    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584714    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584717    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584720    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:20:06.601418 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584722    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584725    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584728    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584730    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584733    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584735    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584738    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584741    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584743    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584746    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584748    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584751    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584754    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584756    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584759    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584761    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584765    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584770    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584774    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:20:06.601954 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584777    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584780    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584783    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584785    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584788    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584792    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584795    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584804    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584807    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584810    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584812    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584815    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584817    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584820    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584823    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584825    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584828    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584831    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584833    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584836    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:20:06.602444 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584838    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584841    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584844    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584846    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584849    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584853    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584856    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584858    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584861    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584865    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584868    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584870    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584873    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:06.584875    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.584944    5958 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.586015    5958 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.586087    5958 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.586293    5958 server.go:1014] "Starting client certificate rotation"
Oct 31 19:20:06.602953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.586902    5958 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.586953    5958 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.586966    5958 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:13:54 +0000 UTC" deadline="2028-05-24 15:27:14.673553943 +0000 UTC"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.587008    5958 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13700h7m8.086547631s"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.588204    5958 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.590737    5958 log.go:25] "Validated CRI v1 runtime API"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.592689    5958 log.go:25] "Validated CRI v1 image API"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.593492    5958 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 19:20:06.603458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.595902    5958 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a06d57c5-f82c-423b-ae6b-871f70db6464:/dev/nvme0n1p3 feb6b71b-649b-43d0-b9c8-8951155b333a:/dev/nvme0n1p4]
Oct 31 19:20:06.603760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.595924    5958 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02d0607a73c2931f3437fb89632fd90c5fe6c6848a5bf6c1e37a7f6dbb208fe8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02d0607a73c2931f3437fb89632fd90c5fe6c6848a5bf6c1e37a7f6dbb208fe8/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/471b77066069c7689da92646cd7e25e50d1c34cae160e4d202b80196a140d6b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/471b77066069c7689da92646cd7e25e50d1c34cae160e4d202b80196a140d6b2/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d9d3a4c5bcca5179e83dddf94b9dc5b99865faea2d117799619db94dd63392f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d9d3a4c5bcca5179e83dddf94b9dc5b99865faea2d117799619db94dd63392f/userdata/shm major:0 minor:483 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba/userdata/shm major:0 minor:461 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe55f2ab79aa2269dbb181052559e7229f18c97afb5d8323aba8303b4c639c36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe55f2ab79aa2269dbb181052559e7229f18c97afb5d8323aba8303b4c639c36/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6/userdata/shm major:0 minor:327 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0312583f-6efa-4d8f-8921-d9deb835d69f/volumes/kubernetes.io~projected/kube-api-access-ghc9v:{mountpoint:/var/lib/kubelet/pods/0312583f-6efa-4d8f-8921-d9deb835d69f/volumes/kubernetes.io~projected/kube-api-access-ghc9v major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/alertmanager/9 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~empty-dir/config-out major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~projected/kube-api-access-xg6f2:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~projected/kube-api-access-xg6f2 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~projected/tls-assets major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/config-volume major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/web-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~projected/kube-api-access-bww5n:{mountpoint:/var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~projected/kube-api-access-bww5n major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22143d8e-6fcb-4563-8989-a29ddefbc053/volumes/kubernetes.io~projected/kube-api-access-ttmdg:{mountpoint:/var/lib/kubelet/pods/22143d8e-6fcb-4563-8989-a29ddefbc053/volumes/kubernetes.io~projected/kube-api-access-ttmdg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf/volumes/kubernetes.io~projected/kube-api-access-x64r5:{mountpoint:/var/lib/kubelet/pods/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf/volumes/kubernetes.io~projected/kube-api-access-x64r5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2825d9bc-c104-43b0-bbe5-acbfd127f71c/volumes/kubernetes.io~projected/kube-api-access-84d8m:{mountpoint:/var/lib/kubelet/pods/2825d9bc-c104-43b0-bbe5-acbfd127f71c/volumes/kubernetes.io~projected/kube-api-access-84d8m major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31c31f22-7c5d-43bb-b554-ad463e4702c5/volumes/kubernetes.io~projected/kube-api-access-gc9qj:{mountpoint:/var/lib/kubelet/pods/31c31f22-7c5d-43bb-b554-ad463e4702c5/volumes/kubernetes.io~projected/kube-api-access-gc9qj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31c31f22-7c5d-43bb-b554-ad463e4702c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/31c31f22-7c5d-43bb-b554-ad463e4702c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39a2e073-9cf5-4472-9ba8-df174e1bb3e0/volumes/kubernetes.io~projected/kube-api-access-5njb9:{mountpoint:/var/lib/kubelet/pods/39a2e073-9cf5-4472-9ba8-df174e1bb3e0/volumes/kubernetes.io~projected/kube-api-access-5njb9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39a2e073-9cf5-4472-9ba8-df174e1bb3e0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/39a2e073-9cf5-4472-9ba8-df174e1bb3e0/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~projected/kube-api-access-88cx4:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~projected/kube-api-access-88cx4 major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~projected/kube-api-access-gsq4c:{mountpoint:/var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~projected/kube-api-access-gsq4c major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~projected/kube-api-access-46f7z:{mountpoint:/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~projected/kube-api-access-46f7z major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56c7eb65-3951-480e-88e2-1858cae684e6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/56c7eb65-3951-480e-88e2-1858cae684e6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e5ab86-50ff-419a-a4da-2c19408eb9ae/volumes/kubernetes.io~projected/kube-api-access-kk7q4:{mountpoint:/var/lib/kubelet/pods/58e5ab86-50ff-419a-a4da-2c19408eb9ae/volumes/kubernetes.io~projected/kube-api-access-kk7q4 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~projected/kube-api-access-dvf4s:{mountpoint:/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~projected/kube-api-access-dvf4s major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/config-reloader/2 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/prometheus/15 major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~projected/kube-api-access-wbg2t:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~projected/kube-api-access-wbg2t major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~projected/tls-assets major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/config major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/web-config major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~projected/kube-api-access-s5c2g:{mountpoint:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~projected/kube-api-access-s5c2g major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a291037-a7e5-4cb8-bfde-a5aec293cbeb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9a291037-a7e5-4cb8-bfde-a5aec293cbeb/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a237adff-6089-4d22-993f-95a9869a9771/volumes/kubernetes.io~projected/kube-api-access-nkmw9:{mountpoint:/var/lib/kubelet/pods/a237adff-6089-4d22-993f-95a9869a9771/volumes/kubernetes.io~projected/kube-api-access-nkmw9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a989c545-d91d-4ca2-9dea-0637a462d204/volumes/kubernetes.io~projected/kube-api-access-kjtq2:{mountpoint:/var/lib/kubelet/pods/a989c545-d91d-4ca2-9dea-0637a462d204/volumes/kubernetes.io~projected/kube-api-access-kjtq2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a989c545-d91d-4ca2-9dea-0637a462d204/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a989c545-d91d-4ca2-9dea-0637a462d204/volumes/kubernetes.io~secret/metrics-certs major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volumes/kubernetes.io~projected/kube-api-access-6bfzk:{mountpoint:/var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volumes/kubernetes.io~projected/kube-api-access-6bfzk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2d2e4e2-5258-4be1-929f-1d92b689c0ff/volumes/kubernetes.io~projected/kube-api-access-drnnf:{mountpoint:/var/lib/kubelet/pods/d2d2e4e2-5258-4be1-929f-1d92b689c0ff/volumes/kubernetes.io~projected/kube-api-access-drnnf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2d2e4e2-5258-4be1-929f-1d92b689c0ff/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d2d2e4e2-5258-4be1-929f-1d92b689c0ff/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~projected/kube-api-access-w7sgm:{mountpoint:/var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~projected/kube-api-access-w7sgm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d92bf693-c264-4e52-84a5-045804009257/volumes/kubernetes.io~projected/kube-api-access-bgx2q:{mountpoint:/var/lib/kubelet/pods/d92bf693-c264-4e52-84a5-045804009257/volumes/kubernetes.io~projected/kube-api-access-bgx2q major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~projected/kube-api-access-dkvnd:{mountpoint:/var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~projected/kube-api-access-dkvnd major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~projected/kube-api-access-5n66g:{mountpoint:/var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~projected/kube-api-access-5n66g major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/abfe9ce727679d9c13a1cccc49a8234f0be2fe231b7876b2d9d04dfcb8dfc90e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d0a4d704d8da43556a5e3be08e48bf8bef46c6036b708063aaa941f0137a2121/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/48bc1e2036ddfda34d91296f66e85889901efc481c1d9f9a00e15b7f66145969/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7078a6c7086b1fef9b5f6459432d33016e165ef637712c150d3c469ce0e1210f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/dbdc69a02d9f99f9b7ecf9b747c776c1b7cd542c61bb1b40c3406c078912fbc1/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/07b7335741e39cfd2d77d921143e81236ebefdf3e2efc1f00846b26deaf0dfee/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1e6799488a7015cb1272dae8fababa7ce13c7988af73d2bb8b5fa50072a53e5e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8e6e7cadeb9a11ce7615045d396d24dc5aac8fb6df0890c41a7c3f35a0069ef2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8cfc018639cbc6c0389147b05d7b0f5241ef4b5e449b0937908599efaba840ed/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0cc2428aefbe9407f7f266a34e0e07751a3385ea3643a1b71147956c158907c6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/b1f0879d09ef2cb1bc6bcb4a9cb494e4068a23232493ad9ca802e45a0fe4f4d5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1dcc6f6c960e55e794b228c874687caad5cc7007d3cf9c826868eb8b3fd6c806/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9299b6d6f9297829926098176641c6eeb1118a9a743e833a4e5b3ceb166b95ef/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/01bff3e465e9835f61645a293bf6090df63523f0eec9ce95ebdc27cf5d2d0c2f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a8c50f7cffa57646dd82cec9fa50e5588895d1e0fef577d1ca8bdf4e00340147/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fdd82a156bd8d56f0352c0d01bf068de3561accd52c34d852a69e24a3b611761/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ef58d9c8efc684157c83614f0e03082afde69e19dfa8c7131ca1da83e92fbf02/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/29c3bd8f1038e71af342457ddb665812ce2a448601327e5ec4f68f22b4fc8f05/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/6d2a9fc9e8341b74ee65651c06deed3fcce56225bf127219114b4a5f2d640f71/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/5b0f9fa3e2e614910ad3e4d2ed8c05053db07b34f2b686ea93fc75c42a00b606/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/539ebce70689989b16901a21b55c7055edb87dd7cef8b5a66af8a7165f40c75d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d7223302bc97201bee6605e8980228538ade5f180229730cfeff1879d74f91a1/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/cadb70c502c0c5b5bdb1255bdb7be47ffa972f2e314285edf82e6f0db336da11/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/ff565f18a5ca0392c9e3a6c9ac384bb33d4107a94fb6490586cea007246161d9/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/eef065b9abddd9a514b792a047767a8693fba5fbd73a1e18bdbba87e805bad39/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b47d01a7d5d1b7415ac1924194810b6037b8a898f2a3376718852da146b73186/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/834dd002e4df1793b07c5d6fa56152c481acdd99065eaf4fce9e779b9089b052/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/6007ea65c848ca5b9fdfc4d3567ee0d0656ab540c78bdb4cb64153de3765ca1b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e9d4df7015144d9902db36fe65045045e90f131f7b36cbc556c6bb362c3db168/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/caa98722c26271cf5447e2fb77ca9e0eedb14d4f245d723ae6e49470e0419118/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d90d4504c5b10a0eb5aa4b35014dff1b7d5ec2713b286817dd1cc4034969402c/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/31142f71314fc00fc8c49689d8c8133043c4603aac3f7efc316507538c0de2cf/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/cb6440a46aaca53a6aee54d4f97ba592e2ddffebb7a4dd51b5ca5ee565387031/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/4d976854413b35c05aa58cfd9ea7331775f4f66a54e0487111154c5c741118b8/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/f34204e4f01bf4348888742f11f27247a549f83561cf6b32e2f36a2739f69f8a/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/b32d3796c59d1512d322662a6ccbd36563d67523d1ae96fdf3d35215570d9a1a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/208ff244f1c9d4f4b84572e84ee2baab139330970d6826011695753d295c5cb6/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/bcda49c899640b16cc021d07c47941c36c8f06227d25ae4ce604f50cb9029d9b/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/8a06698bd11b482b8f02a47b3e03b525217bf323e9ed8868d0c1a3f14e3191bb/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/645e5917b3f9665d147abb8670f7cbb2b133e8951d9736ad6e142548356fcc2b/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/cb67530032cee0105c892d8d8c2cc96a2e1aa396e89aeec67a0cb80ef8e9dc92/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/8c109987d4663ca64defa85123e755af900ae6634969d490d826763deb8827d6/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/48d3f936522e5ab906c85db51c5594caa0782139d88f826d2dc802aff7be2cdc/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/a39e9f6995d6d76bc6d817e01877cc1658ad76913d2ba8da359b9492f689921f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a81bdd0ff16ec3132b59e7e90e0baa665838208fc970d40f0d5ded85f443f359/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/f6a174aff327958754bcc2ab6f53300c8106bf7d2d4f25bde4ed2ce643ffbc63/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/1dc115352247f1f7948545e337180f78549afa2561406a0b97c6f9d77b6fa83d/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/3be3dc6275b90cf0f2d831a7efc814c29120ca5729490383809bb9d20546421e/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/5ed161e0efad985f4251664612a6f7875b1992b113aeeeb8a15da39c1bdac520/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/7e240033b6bbea6750cd8e4fa5eb47869266dcacb43e6a102667249c58337d75/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/f18547fbf16d467fbea9cbd5d01922ce0189bcc0144209482e6128e83e514c43/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/8418e6fd9ba49ebdf57bde81beaa190d07019204456421a7acb1c25a785631b0/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/4f27ef2c6d1a2193f1bfd6382761ddd7a966d9fe1f25126fe39a6650d8a1fba8/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/e669b9884276885912ecbac406ba6ae70e111058e7aa93770692618957f0410f/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/8b18a485c66769963b636ebc1d58c76b28637a4f0dddd0e402abd25071a8b2c9/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/2570ec94e12cdfe323c1dddee854fcd02c60c22f69948751c8c25a1e2cc4223e/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/b89bab3975e0e6cc7e9f230021412809f447e96352fa8722840440fe416f3742/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/710e7eb31e18203460b8b97d209c20759a0ffa888846c29952758f58b2d4f14d/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/dafc672ba7d276a4929da0ca5b5874be45c9b27dc68f599202b8c50e0712dc50/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/65d47bf5d44f4bfa48499f6afb22f05cd3b4dd7bb33cff4b76fae4263f18d87a/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/db03e43990558b395cd94a9163055e9c16b84b989340c51b9bce35f3baf04e37/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/bd1774d47769a090bf1ecdd0b44ce84e1ffdd2afb56d15c9962dc1e25c2e2973/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/f648156611514ad23fc8410175adc512ed89570f3f6a218f21e402ffa55e78b7/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/a1c2437c6bc85c79f291c0efbcf33fefe63f4d868f8fc2f5fc052b9a18f5f9ce/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/d33568da0fd6af90dd631afc1ffd98c9963b4500ca93046ac1c59592ab2d9415/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/bcb776659bc4be30498b7ca4e9985dfd44d544c70d6ae6ae832d81179b526a11/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/d5c5f97a25aa91784c8b8ebdd492aca104c3d202473da23b3c8dd3e38d12bc95/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/6f025b2027655285f8fd81d241ed3e0ad446ce8c6b97add356b24aca7e21b9b5/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/64abb1d92b868c7e58a4320b16d906ee4fe61ab2a51e713c5e7b0f82d7c76a1f/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/03524ff09f50e2e7efcd3ed3135ab4e48664b49a970271270e598aa78dcf0b7f/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/d4016df9a35147315257ed7855c3edadc94b953741e4a5b2ec23a8cf493d8869/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/40fc0ee91d31f04387acb6944661d10eb78620019f4e212a3d7bf05746512f4d/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/40c6abf51a080df3093fd44aae4726c677bef1fa6690a5e7b4645214dbef426c/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/4ebb0c53da03a0e7a0643dc3e338d8b78d353cdc9f59d9edde43219346eff9bb/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/2348c95f4f691d3468b242f9763b9168070bcb711f1614ee7e229bd239693e8d/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/cb022f68857c50e5d5e372b45e1c09955b5b2ea2fedcfa8dd999ecf838c1aa93/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/330dade60bca28ce39f29b4a0c64f1bde6d371c25175c218f28aaded3a21565d/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/7e80360e8517a95325dd20e2b5544005f0db45c0c8e114bf67d7db8e8e176419/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/64903ab6b0454b697824096049636e11099b383fa3472c7dbddd518a4a77131f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-533:{mountpoint:/var/lib/containers/storage/overlay/68ea62a924096d48a881e2794cc216c77916c772c46d90d0df89c14460d459d3/merged major:0 minor:533 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ba15996a7c7d6ca2581675aa9f168287f56b344d47c6fd0fcd75801c44eed068/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a2b9f8a6398a139eadb2e1865c2b130f0a023e0bb981dcda21681cf8d1462df9/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/57e997b217b06fcbfb4d2954612f547f541ffea3431a0fc41ebed0383f75e8ff/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ba324f063c7aee8b88fccb9c0457ac23bf636e53d18c83d151583a8588bda52b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/08359070aa5116ad169cac7ca8f30bac2256db4bf68f6bc41fe689215048b26c/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/5c1da67d146344fa92e049656a4338d3abd9d7a80c08ea1745b7a6305afb0c7c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/83cc43127af99298cd1c2a10135b31ee159544c6e55e9e018fe0509243a565e9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/398b062558601c838592835f36d4df114e8b50a67c3055114f6ababa89d65acf/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 19:20:06.606520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.606029    5958 manager.go:217] Machine: {Timestamp:2026-10-31 19:20:06.605509365 +0000 UTC m=+0.066883981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099904 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec287c06a8d3af57ea3be40edfbf38e0 SystemUUID:ec287c06-a8d3-af57-ea3b-e40edfbf38e0 BootID:3e7d9fa8-dd64-4210-a9a1-507e7fa4fe82 Filesystems:[{Device:/run/containers/storage/overlay-containers/92de927e0a629378141a295dc91d8ae14a196477bebfff63370bb40a5290964c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98cbf1ac280e9df65a5b70b426b0b7afcc25760f7b43e96c1ae7881624610f18/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a291037-a7e5-4cb8-bfde-a5aec293cbeb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:453 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2d2e4e2-5258-4be1-929f-1d92b689c0ff/volumes/kubernetes.io~projected/kube-api-access-drnnf DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d92bf693-c264-4e52-84a5-045804009257/volumes/kubernetes.io~projected/kube-api-access-bgx2q DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:459 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70b67c1d7b86e48db94b63a43320964dcf80edd660eb58c723aa14c3e7a67f27/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b363532112e496e119d47c7e35f537fb4ca24a57ef081557b203d2fe931f2312/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31c31f22-7c5d-43bb-b554-ad463e4702c5/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:458 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~projected/kube-api-access-wbg2t DeviceMajor:0 DeviceMinor:460 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/162fd1bc-c40c-4001-8fa3-8258ed577b45/volumes/kubernetes.io~projected/kube-api-access-bww5n DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:311 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39a2e073-9cf5-4472-9ba8-df174e1bb3e0/volumes/kubernetes.io~projected/kube-api-access-5njb9 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:453 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe55f2ab79aa2269dbb181052559e7229f18c97afb5d8323aba8303b4c639c36/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dc47eda7ddcaeed2a11313730aa5adf663d5e78a58bce81a87df4a9ef42bf47/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08e013dee3f2a32b544b072bf7a0312ae1adee40244a201d18b61afa10097649/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58e5ab86-50ff-419a-a4da-2c19408eb9ae/volumes/kubernetes.io~projected/kube-api-access-kk7q4 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:450 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a989c545-d91d-4ca2-9dea-0637a462d204/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:470 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:384 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~projected/kube-api-access-xg6f2 DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:451 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:453 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0312583f-6efa-4d8f-8921-d9deb835d69f/volumes/kubernetes.io~projected/kube-api-access-ghc9v DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:456 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1be9b7ef8c2ee40042a9c7d063a2ba1167d6132c79670212967d21eac3b4f2b/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-533 DeviceMajor:0 DeviceMinor:533 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d9d3a4c5bcca5179e83dddf94b9dc5b99865faea2d117799619db94dd63392f/userdata/shm DeviceMajor:0 DeviceMinor:483 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~projected/kube-api-access-s5c2g DeviceMajor:0 DeviceMinor:385 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a989c545-d91d-4ca2-9dea-0637a462d204/volumes/kubernetes.io~projected/kube-api-access-kjtq2 DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3689afcfb836a47a15352e801878d5bde9cd41a930d990b74a49ee815f5cf883/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~projected/kube-api-access-46f7z DeviceMajor:0 DeviceMinor:326 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~projected/kube-api-access-w7sgm DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d1c8f6ea67903140292c9301e88cb556d7f5808915a0c2236504e78ce3fbd9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~projected/kube-api-access-dvf4s DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d2d2e4e2-5258-4be1-929f-1d92b689c0ff/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~projected/kube-api-access-88cx4 DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22143d8e-6fcb-4563-8989-a29ddefbc053/volumes/kubernetes.io~projected/kube-api-access-ttmdg DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a237adff-6089-4d22-993f-95a9869a9771/volumes/kubernetes.io~projected/kube-api-access-nkmw9 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/volumes/kubernetes.io~projected/kube-api-access-5n66g DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7107ba5950a10305fc9c74b4f1788e3af061237ec83b40d9a4081fee165f3b23/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~projected/kube-api-access-gsq4c DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02d0607a73c2931f3437fb89632fd90c5fe6c6848a5bf6c1e37a7f6dbb208fe8/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/56c7eb65-3951-480e-88e2-1858cae684e6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:471 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:280 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:325 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/877374f8e0fb95846f4552e7c959a5ce12689e22d6ed7cc27b731822a945ae11/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ec17dff9e328facb4249936d75e790b2463987b895305b586dba75c03ab3965/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/39a2e073-9cf5-4472-9ba8-df174e1bb3e0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:453 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d48cdb47-67b2-47ec-9433-d831d8393f53/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6/userdata/shm DeviceMajor:0 DeviceMinor:327 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4184cf94-99b2-4491-a90f-a523b8f9ba3c/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ac670f323f2ab1336119bdac1f6fd870be64a345958960a5ff2d008a03b39bc/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acccc566ed7755f3aa5ce96fa11dc54149458c70141949bc12647f569032794d/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf/volumes/kubernetes.io~projected/kube-api-access-x64r5 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c9647c81-f8aa-470d-a5c7-02897a93fa31/volumes/kubernetes.io~projected/kube-api-access-6bfzk DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e59612d58f265551a8fb2026c0a9c98d46a79539ea95393dc0252c644ed6d25e/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/471b77066069c7689da92646cd7e25e50d1c34cae160e4d202b80196a140d6b2/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/555bb1b5-4eea-4de4-b4a0-45613823a90b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98bdd4d0-f6c1-4a72-a603-e4928359958d/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31c31f22-7c5d-43bb-b554-ad463e4702c5/volumes/kubernetes.io~projected/kube-api-access-gc9qj DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/volumes/kubernetes.io~projected/kube-api-access-dkvnd DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:455 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2825d9bc-c104-43b0-bbe5-acbfd127f71c/volumes/kubernetes.io~projected/kube-api-access-84d8m DeviceMajor:0 DeviceMinor:474 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:457 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/095005d29c8aa8b8232d46aed7c1cccb8472a3811b6afe205bf8edfd18799137/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:452 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba/userdata/shm DeviceMajor:0 DeviceMinor:461 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08e013dee3f2a32 MacAddress:32:b8:dc:18:c5:3a Speed:10000 Mtu:8901} {Name:095005d29c8aa8b MacAddress:d2:f3:7c:d4:d9:3a Speed:10000 Mtu:8901} {Name:24c8d577dc2f399 MacAddress:ba:4e:2f:6b:18:8b Speed:10000 Mtu:8901} {Name:2ac670f323f2ab1 MacAddress:46:28:e6:90:80:b2 Speed:10000 Mtu:8901} {Name:3689afcfb836a47 MacAddress:2e:23:f1:c6:c9:bc Speed:10000 Mtu:8901} {Name:471b77066069c76 MacAddress:66:8f:c4:43:95:90 Speed:10000 Mtu:8901} {Name:70b67c1d7b86e48 MacAddress:da:d7:57:a9:9f:0b Speed:10000 Mtu:8901} {Name:7107ba5950a1030 MacAddress:8a:27:b9:c8:7f:ea Speed:10000 Mtu:8901} {Name:7d9d3a4c5bcca51 MacAddress:ee:38:d0:7c:12:5f Speed:10000 Mtu:8901} {Name:98cbf1ac280e9df MacAddress:b2:f5:d4:0e:43:66 Speed:10000 Mtu:8901} {Name:acccc566ed7755f MacAddress:4e:5c:e9:44:00:ba Speed:10000 Mtu:8901} {Name:b27115ecaf8d3f5 MacAddress:76:1d:ac:68:55:cc Speed:10000 Mtu:8901} {Name:b36404272e752f9 MacAddress:ea:a3:f8:84:b1:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e5:18:a7:77 Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:bf:ce:9c:e6:9e Speed:0 Mtu:8901} {Name:e1be9b7ef8c2ee4 MacAddress:a2:06:9e:fe:8c:2f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e5:18:a7:77 Speed:0 Mtu:9001} {Name:feaf09c7f9aebe1 MacAddress:32:29:af:3a:17:5c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1e:30:cb:eb:31:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:4d:59:cc:64:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 19:20:06.606520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.606522    5958 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 19:20:06.606865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.606595    5958 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 19:20:06.606924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.606896    5958 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 19:20:06.607114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.606925    5958 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-35.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 19:20:06.607160 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607117    5958 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 19:20:06.607160 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607126    5958 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 19:20:06.607160 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607139    5958 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:20:06.607160 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607153    5958 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:20:06.607271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607262    5958 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 19:20:06.607461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607451    5958 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 19:20:06.607606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607596    5958 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 19:20:06.607638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607634    5958 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 19:20:06.607671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607647    5958 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 19:20:06.607671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607659    5958 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 19:20:06.607671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607670    5958 kubelet.go:405] "Adding apiserver pod source"
Oct 31 19:20:06.607746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.607678    5958 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 19:20:06.609899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.608996    5958 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 19:20:06.609899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.609017    5958 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:20:06.610127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.609925    5958 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 19:20:06.613419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613404    5958 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 19:20:06.613419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613419    5958 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:20:06.613553 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613530    5958 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 19:20:06.613756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613744    5958 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 19:20:06.613792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613775    5958 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 19:20:06.613888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613879    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 19:20:06.613921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613896    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 19:20:06.613921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613903    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 19:20:06.613921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613910    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 19:20:06.613921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613920    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613929    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613936    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613941    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613948    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613954    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613969    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.613978    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.614003    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 19:20:06.614023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.614012    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 19:20:06.617872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.617860    5958 server.go:1263] "Started kubelet"
Oct 31 19:20:06.617924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.617903    5958 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 19:20:06.618050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.617972    5958 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 19:20:06.618120 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.618077    5958 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 19:20:06.618437 ip-10-0-138-35 systemd[1]: Started Kubernetes Kubelet.
Oct 31 19:20:06.618610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.618499    5958 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 19:20:06.619417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.619378    5958 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 19:20:06.622979 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.622961    5958 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 19:20:06.622979 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.622972    5958 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.623672    5958 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.623812    5958 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.623107    5958 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:55 +0000 UTC" deadline="2028-07-31 18:41:59.18371775 +0000 UTC"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.625560    5958 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15335h21m52.55818108s"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.625834    5958 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.627172    5958 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-35.us-east-2.compute.internal\" not found"
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.627919    5958 factory.go:55] Registering systemd factory
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.627940    5958 factory.go:223] Registration of the systemd container factory successfully
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.628196    5958 factory.go:153] Registering CRI-O factory
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.628209    5958 factory.go:223] Registration of the crio container factory successfully
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.628268    5958 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.628286    5958 factory.go:103] Registering Raw factory
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.628298    5958 manager.go:1199] Started watching for new ooms in manager
Oct 31 19:20:06.628847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.628590    5958 manager.go:319] Starting recovery of all containers
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631785    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" volumeName="kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631829    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bb1b5-4eea-4de4-b4a0-45613823a90b" volumeName="kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631845    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9647c81-f8aa-470d-a5c7-02897a93fa31" volumeName="kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631856    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f300c7e3-5d90-4bc7-bd34-9d8e743d9f27" volumeName="kubernetes.io/projected/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-api-access-dkvnd" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631869    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631882    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/projected/4184cf94-99b2-4491-a90f-a523b8f9ba3c-kube-api-access-88cx4" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631895    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631910    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631926    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631939    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631951    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9647c81-f8aa-470d-a5c7-02897a93fa31" volumeName="kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-env-overrides" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631963    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" volumeName="kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631976    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.631990    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632006    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9647c81-f8aa-470d-a5c7-02897a93fa31" volumeName="kubernetes.io/secret/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632038    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2d2e4e2-5258-4be1-929f-1d92b689c0ff" volumeName="kubernetes.io/projected/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-kube-api-access-drnnf" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632051    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632066    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f300c7e3-5d90-4bc7-bd34-9d8e743d9f27" volumeName="kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632080    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632093    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bb1b5-4eea-4de4-b4a0-45613823a90b" volumeName="kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632105    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632116    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-client-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632138    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a291037-a7e5-4cb8-bfde-a5aec293cbeb" volumeName="kubernetes.io/secret/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-agent-certs" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632151    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9647c81-f8aa-470d-a5c7-02897a93fa31" volumeName="kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632166    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2" volumeName="kubernetes.io/configmap/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632182    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162fd1bc-c40c-4001-8fa3-8258ed577b45" volumeName="kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632195    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c31f22-7c5d-43bb-b554-ad463e4702c5" volumeName="kubernetes.io/secret/31c31f22-7c5d-43bb-b554-ad463e4702c5-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632207    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a2e073-9cf5-4472-9ba8-df174e1bb3e0" volumeName="kubernetes.io/projected/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-kube-api-access-5njb9" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632225    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632239    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a989c545-d91d-4ca2-9dea-0637a462d204" volumeName="kubernetes.io/projected/a989c545-d91d-4ca2-9dea-0637a462d204-kube-api-access-kjtq2" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632252    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f300c7e3-5d90-4bc7-bd34-9d8e743d9f27" volumeName="kubernetes.io/empty-dir/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-volume-directive-shadow" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632267    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632299    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2825d9bc-c104-43b0-bbe5-acbfd127f71c" volumeName="kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632315    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632329    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632342    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632357    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d92bf693-c264-4e52-84a5-045804009257" volumeName="kubernetes.io/projected/d92bf693-c264-4e52-84a5-045804009257-kube-api-access-bgx2q" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632373    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162fd1bc-c40c-4001-8fa3-8258ed577b45" volumeName="kubernetes.io/projected/162fd1bc-c40c-4001-8fa3-8258ed577b45-kube-api-access-bww5n" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632387    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/empty-dir/4184cf94-99b2-4491-a90f-a523b8f9ba3c-tmp" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632400    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632416    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9647c81-f8aa-470d-a5c7-02897a93fa31" volumeName="kubernetes.io/projected/c9647c81-f8aa-470d-a5c7-02897a93fa31-kube-api-access-6bfzk" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632432    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2" volumeName="kubernetes.io/projected/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-kube-api-access-5n66g" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632446    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632460    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632471    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e5ab86-50ff-419a-a4da-2c19408eb9ae" volumeName="kubernetes.io/projected/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kube-api-access-kk7q4" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632499    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632511    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2d2e4e2-5258-4be1-929f-1d92b689c0ff" volumeName="kubernetes.io/configmap/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-config-volume" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632524    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632536    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632564    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162fd1bc-c40c-4001-8fa3-8258ed577b45" volumeName="kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632577    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22143d8e-6fcb-4563-8989-a29ddefbc053" volumeName="kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-cni-binary-copy" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632590    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/configmap/4184cf94-99b2-4491-a90f-a523b8f9ba3c-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632601    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632612    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632624    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632636    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0312583f-6efa-4d8f-8921-d9deb835d69f" volumeName="kubernetes.io/projected/0312583f-6efa-4d8f-8921-d9deb835d69f-kube-api-access-ghc9v" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632657    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632669    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48cdb47-67b2-47ec-9433-d831d8393f53" volumeName="kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-tuned" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632679    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2d2e4e2-5258-4be1-929f-1d92b689c0ff" volumeName="kubernetes.io/empty-dir/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-tmp-dir" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632690    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f300c7e3-5d90-4bc7-bd34-9d8e743d9f27" volumeName="kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632700    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39a2e073-9cf5-4472-9ba8-df174e1bb3e0" volumeName="kubernetes.io/secret/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-cert" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632713    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bb1b5-4eea-4de4-b4a0-45613823a90b" volumeName="kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632726    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632737    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a237adff-6089-4d22-993f-95a9869a9771" volumeName="kubernetes.io/projected/a237adff-6089-4d22-993f-95a9869a9771-kube-api-access-nkmw9" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632749    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632762    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/empty-dir/98bdd4d0-f6c1-4a72-a603-e4928359958d-audit-log" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632775    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48cdb47-67b2-47ec-9433-d831d8393f53" volumeName="kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-tmp" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632787    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f300c7e3-5d90-4bc7-bd34-9d8e743d9f27" volumeName="kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632799    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632811    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632822    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c7eb65-3951-480e-88e2-1858cae684e6" volumeName="kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632834    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632846    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/projected/98bdd4d0-f6c1-4a72-a603-e4928359958d-kube-api-access-s5c2g" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632861    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bdd4d0-f6c1-4a72-a603-e4928359958d" volumeName="kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632873    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2" volumeName="kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632884    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632897    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a989c545-d91d-4ca2-9dea-0637a462d204" volumeName="kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632910    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2d2e4e2-5258-4be1-929f-1d92b689c0ff" volumeName="kubernetes.io/secret/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-metrics-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632923    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2" volumeName="kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632936    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632948    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632961    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22143d8e-6fcb-4563-8989-a29ddefbc053" volumeName="kubernetes.io/projected/22143d8e-6fcb-4563-8989-a29ddefbc053-kube-api-access-ttmdg" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632973    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632985    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.632998    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0312583f-6efa-4d8f-8921-d9deb835d69f" volumeName="kubernetes.io/configmap/0312583f-6efa-4d8f-8921-d9deb835d69f-iptables-alerter-script" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633011    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633022    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633032    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" volumeName="kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-binary-copy" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633044    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633056    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a237adff-6089-4d22-993f-95a9869a9771" volumeName="kubernetes.io/configmap/a237adff-6089-4d22-993f-95a9869a9771-serviceca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633069    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22143d8e-6fcb-4563-8989-a29ddefbc053" volumeName="kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-daemon-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633080    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" volumeName="kubernetes.io/projected/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-kube-api-access-x64r5" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633091    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-grpc-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633104    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bb1b5-4eea-4de4-b4a0-45613823a90b" volumeName="kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633117    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633128    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a291037-a7e5-4cb8-bfde-a5aec293cbeb" volumeName="kubernetes.io/configmap/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-konnectivity-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633139    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633152    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c31f22-7c5d-43bb-b554-ad463e4702c5" volumeName="kubernetes.io/projected/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-api-access-gc9qj" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633165    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633177    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d92bf693-c264-4e52-84a5-045804009257" volumeName="kubernetes.io/empty-dir/d92bf693-c264-4e52-84a5-045804009257-tmp-dir" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633188    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633200    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bb1b5-4eea-4de4-b4a0-45613823a90b" volumeName="kubernetes.io/empty-dir/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-textfile" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633212    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633227    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633238    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555bb1b5-4eea-4de4-b4a0-45613823a90b" volumeName="kubernetes.io/projected/555bb1b5-4eea-4de4-b4a0-45613823a90b-kube-api-access-gsq4c" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633252    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633263    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633276    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633288    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633301    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="162fd1bc-c40c-4001-8fa3-8258ed577b45" volumeName="kubernetes.io/configmap/162fd1bc-c40c-4001-8fa3-8258ed577b45-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633314    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c31f22-7c5d-43bb-b554-ad463e4702c5" volumeName="kubernetes.io/configmap/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633326    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633355    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d48cdb47-67b2-47ec-9433-d831d8393f53" volumeName="kubernetes.io/projected/d48cdb47-67b2-47ec-9433-d831d8393f53-kube-api-access-w7sgm" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633368    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" volumeName="kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633380    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c31f22-7c5d-43bb-b554-ad463e4702c5" volumeName="kubernetes.io/empty-dir/31c31f22-7c5d-43bb-b554-ad463e4702c5-data-volume" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633392    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4184cf94-99b2-4491-a90f-a523b8f9ba3c" volumeName="kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633403    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" volumeName="kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633414    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f300c7e3-5d90-4bc7-bd34-9d8e743d9f27" volumeName="kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-metrics-client-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633425    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633437    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adc602-d71a-4b6c-be3e-5033bd5d6393" volumeName="kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633449    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633462    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d1411a2-289b-443e-ba7c-663c8f931635" volumeName="kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config" seLinuxMountContext=""
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633474    5958 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 19:20:06.633695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.633482    5958 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 19:20:06.640051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.639885    5958 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:20:06.642322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.642238    5958 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:20:06.643684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.643665    5958 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 19:20:06.647513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.645808    5958 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:20:06.653474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.653424    5958 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 19:20:06.655304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.655273    5958 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 19:20:06.655304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.655303    5958 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 19:20:06.655474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.655333    5958 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 19:20:06.655474 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.655377    5958 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 19:20:06.663760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.663729    5958 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:20:06.694318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.694268    5958 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="217c947e5ecf63095f13b67ee0ad52f898e3a15ada610bab35431cac80f73682" exitCode=0
Oct 31 19:20:06.694318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.694311    5958 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="71f86b18c9333db72682eafb2be1d80d4e0053ee32ed3f3a35e04c44c73a526c" exitCode=0
Oct 31 19:20:06.694318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.694319    5958 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="4f6ecd3e7a2e1ad99fb384563bdbfaa37744a00d1470173bf2300fe56418ea13" exitCode=0
Oct 31 19:20:06.694318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.694326    5958 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="5cd76d59b7e8552920586fffa957cf19d8e00e752558dc007a5999e60feda937" exitCode=0
Oct 31 19:20:06.694318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.694332    5958 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="65298ab3d58a1a6b8d98bdf46452e8354e9931802580fa2d701f3abc9b968bda" exitCode=0
Oct 31 19:20:06.694679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.694337    5958 generic.go:356] "Generic (PLEG): container finished" podID="26ccae21-e8ff-4b3a-b613-7fdd52aeacbf" containerID="b5cdd9bbab1f147989757d0bcf7081c21000f34655845368c5b88fa878f77a65" exitCode=0
Oct 31 19:20:06.705493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.705459    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6" exitCode=0
Oct 31 19:20:06.707737 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.707710    5958 generic.go:356] "Generic (PLEG): container finished" podID="555bb1b5-4eea-4de4-b4a0-45613823a90b" containerID="c6cc23202cf636085e4e2c7c80a3220cd776edfebe15e64ea2612c03af521f94" exitCode=0
Oct 31 19:20:06.721591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.721526    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e" exitCode=0
Oct 31 19:20:06.740319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.740235    5958 generic.go:356] "Generic (PLEG): container finished" podID="b12c7c191df1c0e91dfcc512de902352" containerID="c34a89b014166d1551dd958e877830bd73529594ef4a53692a3847879f7e3730" exitCode=0
Oct 31 19:20:06.755491 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.755458    5958 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 19:20:06.761246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.761222    5958 manager.go:324] Recovery completed
Oct 31 19:20:06.789551 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789512    5958 cpu_manager.go:226] "Starting" policy="none"
Oct 31 19:20:06.789551 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789534    5958 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 19:20:06.789713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789571    5958 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 19:20:06.789768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789741    5958 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 19:20:06.789798 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789769    5958 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 19:20:06.789850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789799    5958 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 19:20:06.789850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789806    5958 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 19:20:06.789850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789815    5958 policy_none.go:50] "Start"
Oct 31 19:20:06.789850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789831    5958 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 19:20:06.789850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789846    5958 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 19:20:06.789992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789982    5958 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 19:20:06.790021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789992    5958 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 19:20:06.790021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.789998    5958 policy_none.go:44] "Start"
Oct 31 19:20:06.795808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.795759    5958 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 19:20:06.800344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.800325    5958 manager.go:341] "Starting Device Plugin manager"
Oct 31 19:20:06.800881 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.800355    5958 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 19:20:06.800881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.800878    5958 server.go:87] "Starting device plugin registration server"
Oct 31 19:20:06.801177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801161    5958 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 19:20:06.801240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801177    5958 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 19:20:06.801349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801322    5958 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 19:20:06.801686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801450    5958 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 19:20:06.801686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801498    5958 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 19:20:06.801686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801506    5958 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 19:20:06.801686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.801568    5958 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:20:06.801487158Z","UUID":"8fd94d21-e140-4839-8f35-2181aa21f62a","Handler":null,"Name":"","Endpoint":""}
Oct 31 19:20:06.803512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.803484    5958 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 19:20:06.803646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.803519    5958 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 19:20:06.803688 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.803648    5958 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 19:20:06.901794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.901759    5958 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:20:06.906465 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.906437    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:20:06.906638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.906479    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:20:06.906638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.906494    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-35.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:20:06.906638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.906602    5958 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.924808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.924777    5958 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.924942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.924862    5958 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.956511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.956440    5958 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"]
Oct 31 19:20:06.957100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957067    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"a1d262f6c678a43aaab0402c233dbed3","Type":"ContainerStarted","Data":"6499f0dbcb841ddbe104577a3c0f315147d57bf9a7e882a90ee2c0b18e5a40da"}
Oct 31 19:20:06.957142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957107    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"a1d262f6c678a43aaab0402c233dbed3","Type":"ContainerStarted","Data":"1b05390fd8377e8820a4922a68f3490058120fc24f5ef4de20bf037b9a6b202e"}
Oct 31 19:20:06.957211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957195    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.957286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957223    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"b12c7c191df1c0e91dfcc512de902352","Type":"ContainerStarted","Data":"ea8716b12c22b63567e5af2431200dda43bbb944abc4e469762ab6663cad8419"}
Oct 31 19:20:06.957286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957236    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"b12c7c191df1c0e91dfcc512de902352","Type":"ContainerDied","Data":"c34a89b014166d1551dd958e877830bd73529594ef4a53692a3847879f7e3730"}
Oct 31 19:20:06.957286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957249    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal" event={"ID":"b12c7c191df1c0e91dfcc512de902352","Type":"ContainerStarted","Data":"c2eaa2ee14a71dac73546a8d38b42f15e934e4393763eaf3b198566c10b6745f"}
Oct 31 19:20:06.957462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.957300    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.973180 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.973156    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:20:06.973317 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.973215    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:06.973854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:06.973834    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:20:06.973920 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:06.973884    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.044233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.044142    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1d262f6c678a43aaab0402c233dbed3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"a1d262f6c678a43aaab0402c233dbed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.044233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.044182    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.044233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.044198    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.144785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.144747    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.144785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.144784    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1d262f6c678a43aaab0402c233dbed3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"a1d262f6c678a43aaab0402c233dbed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.144986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.144802    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.144986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.144846    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.144986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.144867    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b12c7c191df1c0e91dfcc512de902352-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"b12c7c191df1c0e91dfcc512de902352\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.144986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.144877    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1d262f6c678a43aaab0402c233dbed3-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" (UID: \"a1d262f6c678a43aaab0402c233dbed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.608640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.608597    5958 apiserver.go:52] "Watching apiserver"
Oct 31 19:20:07.625083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.625045    5958 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 19:20:07.625812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.625782    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79fdb78f65-nscml","openshift-multus/network-metrics-daemon-bfhh2","openshift-ovn-kubernetes/ovnkube-node-9njm5","openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl","openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw","openshift-monitoring/prometheus-operator-9cd89b74c-t5chn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2","openshift-monitoring/metrics-server-674ffd47d7-5p6sd","openshift-multus/multus-jv8rs","openshift-monitoring/alertmanager-main-0","openshift-image-registry/node-ca-6qz8h","openshift-ingress-canary/ingress-canary-rrq6l","openshift-multus/multus-additional-cni-plugins-qswht","openshift-insights/insights-runtime-extractor-nx6db","openshift-dns/dns-default-vxmdl","openshift-network-diagnostics/network-check-target-9xhcv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal","openshift-console/console-6c7bffb8dc-997rs","openshift-dns/node-resolver-w6g8x","kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal","kube-system/konnectivity-agent-vgtp6","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-bz76x","openshift-cluster-node-tuning-operator/tuned-7w6mv","openshift-monitoring/node-exporter-l2kfm","openshift-monitoring/thanos-querier-b4c88497b-g2wmt","openshift-network-operator/iptables-alerter-9wm8m"]
Oct 31 19:20:07.641900 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.641867    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 19:20:07.641900 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.641866    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcmmt\""
Oct 31 19:20:07.641900 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.641880    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cs8fb\""
Oct 31 19:20:07.642171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642005    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rs77w\""
Oct 31 19:20:07.642171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642088    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gfbm\""
Oct 31 19:20:07.642528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642506    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 19:20:07.642662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642578    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 19:20:07.642662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642638    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.642760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642702    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 19:20:07.642760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642713    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 19:20:07.642760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642723    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.642893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642639    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.642893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642786    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.642893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642842    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 19:20:07.642893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642849    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 19:20:07.643077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642897    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.643077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642927    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.643077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.642970    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 19:20:07.643077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.643038    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.643433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.643410    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.643699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.643685    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.644256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.644236    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 19:20:07.646226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646204    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-system-cni-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646231    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cnibin\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646252    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646325    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646410    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646446    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646468    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64r5\" (UniqueName: \"kubernetes.io/projected/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-kube-api-access-x64r5\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646594    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-binary-copy\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646614    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.646739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.646723    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.667667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.667642    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64r5\" (UniqueName: \"kubernetes.io/projected/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-kube-api-access-x64r5\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.729203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.729169    5958 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 19:20:07.738135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738102    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 19:20:07.738135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738102    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 19:20:07.738612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738586    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5xjgc\""
Oct 31 19:20:07.738612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738588    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 19:20:07.738780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738628    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 19:20:07.738780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738636    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 19:20:07.738780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738725    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rl85m\""
Oct 31 19:20:07.738955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738792    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 19:20:07.738955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738814    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 19:20:07.738955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738872    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 19:20:07.738955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738924    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 19:20:07.739199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.738951    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 19:20:07.739343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739300    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 19:20:07.739406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739379    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrxc5\""
Oct 31 19:20:07.739506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739413    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 19:20:07.739599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739510    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 19:20:07.739599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739533    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbgxr\""
Oct 31 19:20:07.739599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739581    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7xwb\""
Oct 31 19:20:07.739747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739426    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 19:20:07.739747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739667    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\""
Oct 31 19:20:07.739747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.739687    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-w7nnm\""
Oct 31 19:20:07.740193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.740168    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 19:20:07.740275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.740195    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxmqk\""
Oct 31 19:20:07.740275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.740207    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mxbfw\""
Oct 31 19:20:07.740275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.740209    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 19:20:07.740952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.740938    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 19:20:07.741124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741093    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 19:20:07.741124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741100    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 19:20:07.741124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741141    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dvsqv\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741214    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741278    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741288    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741302    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741311    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741302    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 19:20:07.741345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741331    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741363    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4405tprkifkut\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741503    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741564    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7nlh\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741598    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbst6\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741643    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741650    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741689    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741697    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741691    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741720    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741724    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741736    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ao4lshlj74aes\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741730    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741762    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dbvnt\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741774    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741793    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kpnvw\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741886    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwg2q\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741901    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 19:20:07.741948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741940    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741967    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742011    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dw4k4\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.741970    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5n2nlk2eihv8p\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742096    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nkp5g\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742111    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742177    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742196    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742202    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742222    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742228    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:20:07.742727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.742262    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w5n5s\""
Oct 31 19:20:07.743740 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.743627    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.743907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.743849    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 19:20:07.743992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.743977    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 19:20:07.744052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744028    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 19:20:07.744052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744029    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.744141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744087    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 19:20:07.744141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744098    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.744141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744129    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.744141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744131    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.744433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744181    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.744433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744207    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.744433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744383    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 19:20:07.744433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744412    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.744735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744486    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.744735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744589    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.744735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744633    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.744735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744693    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744740    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744749    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744752    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744764    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744784    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744788    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744793    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744809    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 19:20:07.744960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.744831    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 19:20:07.745408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.745395    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.745535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.745523    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.746778 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.746761    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c31f22-7c5d-43bb-b554-ad463e4702c5-data-volume\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.746860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.746788    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.746860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.746811    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0312583f-6efa-4d8f-8921-d9deb835d69f-host-slash\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.746860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.746838    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.746983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.746911    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c31f22-7c5d-43bb-b554-ad463e4702c5-data-volume\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.746983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.746951    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.747072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747018    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-systemd-units\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.747072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747051    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.747255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747072    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.747255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747080    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5njb9\" (UniqueName: \"kubernetes.io/projected/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-kube-api-access-5njb9\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:20:07.747255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747112    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 19:20:07.747255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747163    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-tls\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.747520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747251    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.747520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747301    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 19:20:07.747520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747410    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.747520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747417    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-tls\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.747755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747592    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.747755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747751    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-accelerators-collector-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.747866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747786    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.747866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747806    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.747866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747824    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5c2g\" (UniqueName: \"kubernetes.io/projected/98bdd4d0-f6c1-4a72-a603-e4928359958d-kube-api-access-s5c2g\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.747866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747849    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-netns\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.748039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747877    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-node-log\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.748039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747902    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-k8s-cni-cncf-io\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.748039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747947    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-multus\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.748039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.747984    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.748039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748012    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-dbus\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748042    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748076    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748080    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-textfile\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748139    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-textfile\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748145    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748187    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.748216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748204    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748218    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a237adff-6089-4d22-993f-95a9869a9771-host\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748245    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748282    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0312583f-6efa-4d8f-8921-d9deb835d69f-iptables-alerter-script\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748286    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748341    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748343    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-run\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.748426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748390    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.748659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748430    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-script-lib\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.748659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0312583f-6efa-4d8f-8921-d9deb835d69f-iptables-alerter-script\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.748659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748487    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.748659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748533    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-cni-binary-copy\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.748659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748574    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.748659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748625    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748670    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748708    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748712    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-cni-binary-copy\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748732    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748733    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-script-lib\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748750    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-konnectivity-ca\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748783    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748798    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-kubelet-config\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748815    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.748836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748830    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748858    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748874    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc9v\" (UniqueName: \"kubernetes.io/projected/0312583f-6efa-4d8f-8921-d9deb835d69f-kube-api-access-ghc9v\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748890    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-sys\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748889    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748907    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748905    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d92bf693-c264-4e52-84a5-045804009257-hosts-file\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.748950    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749046    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749074    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749102    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.749226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749152    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749830    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-etc-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749852    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-konnectivity-ca\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749921    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750038    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749133    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a989c545-d91d-4ca2-9dea-0637a462d204-metrics-certs\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.750165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750146    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.750593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.749933    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.750593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750248    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-netd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.750593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750345    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.750593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750408    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.750593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750490    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.750815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750653    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.750815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750708    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.750815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750718    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.750815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750775    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7q4\" (UniqueName: \"kubernetes.io/projected/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kube-api-access-kk7q4\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.750815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750790    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.750815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.750806    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.751100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751071    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/162fd1bc-c40c-4001-8fa3-8258ed577b45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.751152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751114    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a237adff-6089-4d22-993f-95a9869a9771-serviceca\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.751203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751147    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.751203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751164    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-systemd\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.751341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751194    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-system-cni-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.751341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751265    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-metrics-server-audit-profiles\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.751341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751314    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-slash\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.751481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751350    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/162fd1bc-c40c-4001-8fa3-8258ed577b45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.751481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751347    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-env-overrides\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.751481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751395    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfzk\" (UniqueName: \"kubernetes.io/projected/c9647c81-f8aa-470d-a5c7-02897a93fa31-kube-api-access-6bfzk\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.751481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751423    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-hostroot\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.751481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751473    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751518    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751590    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-kubelet\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751625    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751658    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-system-cni-dir\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751665    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751665    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-metrics-server-audit-profiles\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.751735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751704    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n66g\" (UniqueName: \"kubernetes.io/projected/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-kube-api-access-5n66g\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.752089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751764    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.752089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.751849    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cx4\" (UniqueName: \"kubernetes.io/projected/4184cf94-99b2-4491-a90f-a523b8f9ba3c-kube-api-access-88cx4\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.752176 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752119    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a237adff-6089-4d22-993f-95a9869a9771-serviceca\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.752240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752219    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-registration-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.752352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752259    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-kubelet\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.752352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752310    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.752452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752366    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmdg\" (UniqueName: \"kubernetes.io/projected/22143d8e-6fcb-4563-8989-a29ddefbc053-kube-api-access-ttmdg\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.752452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752405    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.752452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752430    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.752641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752603    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.752852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752834    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.752933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752851    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.752933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752868    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-env-overrides\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.752933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752891    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsq4c\" (UniqueName: \"kubernetes.io/projected/555bb1b5-4eea-4de4-b4a0-45613823a90b-kube-api-access-gsq4c\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.752933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752927    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-conf\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.753145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.752986    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.753189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753156    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7sgm\" (UniqueName: \"kubernetes.io/projected/d48cdb47-67b2-47ec-9433-d831d8393f53-kube-api-access-w7sgm\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.753236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753191    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.753279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753262    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-system-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.753321 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753292    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-netns\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.753364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753322    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-bin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.753412 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753359    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.753412 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753392    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkmw9\" (UniqueName: \"kubernetes.io/projected/a237adff-6089-4d22-993f-95a9869a9771-kube-api-access-nkmw9\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.753504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753425    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-client-certs\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.753504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753479    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.753614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753505    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.753614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753564    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.753707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753624    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-metrics-client-ca\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.753707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753664    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.753796 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753706    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31c31f22-7c5d-43bb-b554-ad463e4702c5-crio-socket\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.753796 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753760    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98bdd4d0-f6c1-4a72-a603-e4928359958d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.754060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.753965    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-client-certs\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.754175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754108    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98bdd4d0-f6c1-4a72-a603-e4928359958d-audit-log\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.754244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754176    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/555bb1b5-4eea-4de4-b4a0-45613823a90b-metrics-client-ca\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.754306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754259    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.754358 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754325    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.754358 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754330    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98bdd4d0-f6c1-4a72-a603-e4928359958d-audit-log\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.754461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754380    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-etc-kubernetes\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.754461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754416    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.754461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754435    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtq2\" (UniqueName: \"kubernetes.io/projected/a989c545-d91d-4ca2-9dea-0637a462d204-kube-api-access-kjtq2\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:20:07.754634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754511    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.754634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754593    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-var-lib-kubelet\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.754737 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754708    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.754788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754762    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.754974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754929    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.754974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.754968    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.755183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755030    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.755401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755350    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/162fd1bc-c40c-4001-8fa3-8258ed577b45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.755483 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755441    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755691    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755690    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755694    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755745    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-modprobe-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755869    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cnibin\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755948    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-systemd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.755989    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756046    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26ccae21-e8ff-4b3a-b613-7fdd52aeacbf-cnibin\") pod \"multus-additional-cni-plugins-qswht\" (UID: \"26ccae21-e8ff-4b3a-b613-7fdd52aeacbf\") " pod="openshift-multus/multus-additional-cni-plugins-qswht"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756228    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756285    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756290    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756328    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756338    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756446    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.756586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756567    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756625    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-bin\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756651    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756659    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvnd\" (UniqueName: \"kubernetes.io/projected/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-api-access-dkvnd\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756687    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756755    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-config-volume\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756791    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgx2q\" (UniqueName: \"kubernetes.io/projected/d92bf693-c264-4e52-84a5-045804009257-kube-api-access-bgx2q\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756823    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-device-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756851    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-tuned\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756876    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756909    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756938    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovn-node-metrics-cert\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756968    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.756996    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-tmp-dir\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757020    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757048    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757075    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-lib-modules\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757127    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-config-volume\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757128    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31c31f22-7c5d-43bb-b554-ad463e4702c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.757401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757283    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/162fd1bc-c40c-4001-8fa3-8258ed577b45-kube-api-access-bww5n\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757435    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31c31f22-7c5d-43bb-b554-ad463e4702c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757509    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757605    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-tmp-dir\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757687    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56c7eb65-3951-480e-88e2-1858cae684e6-original-pull-secret\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757775    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovn-node-metrics-cert\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757820    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnnf\" (UniqueName: \"kubernetes.io/projected/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-kube-api-access-drnnf\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757845    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757879    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-tuned\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757902    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.757982    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758066    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758074    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758129    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-tmp\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.758175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758166    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-log-socket\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.758728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758196    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-config\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.758728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758211    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d48cdb47-67b2-47ec-9433-d831d8393f53-tmp\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.758728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758604    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.758728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758643    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-metrics-tls\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.758728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758719    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758772    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758807    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9qj\" (UniqueName: \"kubernetes.io/projected/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-api-access-gc9qj\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758838    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9647c81-f8aa-470d-a5c7-02897a93fa31-ovnkube-config\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758848    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758875    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-conf-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758896    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-metrics-tls\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.758910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758896    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758939    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-kubernetes\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.758999    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759014    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-daemon-config\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759053    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759068    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759095    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-sys-fs\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759136    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysconfig\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759138    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759163    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-agent-certs\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759188    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759243    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-host\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759246    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.759255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759257    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-daemon-config\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759289    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759324    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759368    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4184cf94-99b2-4491-a90f-a523b8f9ba3c-metrics-client-ca\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759397    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a291037-a7e5-4cb8-bfde-a5aec293cbeb-agent-certs\") pod \"konnectivity-agent-vgtp6\" (UID: \"9a291037-a7e5-4cb8-bfde-a5aec293cbeb\") " pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759416    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-var-lib-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759444    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759504    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759563    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-cert\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759607    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4184cf94-99b2-4491-a90f-a523b8f9ba3c-metrics-client-ca\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759615    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759618    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759653    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759684    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-grpc-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759697    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759708    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759740    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-wtmp\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759772    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-cert\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759775    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.759790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759791    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-sys\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759808    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759855    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759907    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4184cf94-99b2-4491-a90f-a523b8f9ba3c-tmp\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759941    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-multus-certs\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759969    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92bf693-c264-4e52-84a5-045804009257-tmp-dir\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759973    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4184cf94-99b2-4491-a90f-a523b8f9ba3c-secret-grpc-tls\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.759995    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-socket-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760007    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760026    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4184cf94-99b2-4491-a90f-a523b8f9ba3c-tmp\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760058    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-client-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760076    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760147    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760171    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-socket-dir-parent\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760206    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92bf693-c264-4e52-84a5-045804009257-tmp-dir\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760220    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-client-ca-bundle\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760258    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-root\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760288    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760318    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-tls\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760403    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760420    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-ovn\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760456    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-cnibin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760478    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760576    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.760652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.760588    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98bdd4d0-f6c1-4a72-a603-e4928359958d-secret-metrics-server-tls\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.763216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.763200    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.764510 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.764493    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.766674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.766660    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:20:07.766725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.766689    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:20:07.766725 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:07.766709    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.766784 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:07.766723    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal"
Oct 31 19:20:07.769632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.769615    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5c2g\" (UniqueName: \"kubernetes.io/projected/98bdd4d0-f6c1-4a72-a603-e4928359958d-kube-api-access-s5c2g\") pod \"metrics-server-674ffd47d7-5p6sd\" (UID: \"98bdd4d0-f6c1-4a72-a603-e4928359958d\") " pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:07.770127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.770111    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsq4c\" (UniqueName: \"kubernetes.io/projected/555bb1b5-4eea-4de4-b4a0-45613823a90b-kube-api-access-gsq4c\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.770533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.770519    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cx4\" (UniqueName: \"kubernetes.io/projected/4184cf94-99b2-4491-a90f-a523b8f9ba3c-kube-api-access-88cx4\") pod \"thanos-querier-b4c88497b-g2wmt\" (UID: \"4184cf94-99b2-4491-a90f-a523b8f9ba3c\") " pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:07.770587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.770532    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmdg\" (UniqueName: \"kubernetes.io/projected/22143d8e-6fcb-4563-8989-a29ddefbc053-kube-api-access-ttmdg\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.770711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.770695    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7q4\" (UniqueName: \"kubernetes.io/projected/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kube-api-access-kk7q4\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.771870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.771832    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") pod \"prometheus-k8s-0\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:07.771870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.771849    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") pod \"console-79fdb78f65-nscml\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") " pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:07.772348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.772311    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc9v\" (UniqueName: \"kubernetes.io/projected/0312583f-6efa-4d8f-8921-d9deb835d69f-kube-api-access-ghc9v\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.772348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.772317    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfzk\" (UniqueName: \"kubernetes.io/projected/c9647c81-f8aa-470d-a5c7-02897a93fa31-kube-api-access-6bfzk\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.772494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.772477    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5njb9\" (UniqueName: \"kubernetes.io/projected/39a2e073-9cf5-4472-9ba8-df174e1bb3e0-kube-api-access-5njb9\") pod \"ingress-canary-rrq6l\" (UID: \"39a2e073-9cf5-4472-9ba8-df174e1bb3e0\") " pod="openshift-ingress-canary/ingress-canary-rrq6l"
Oct 31 19:20:07.772825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.772810    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtq2\" (UniqueName: \"kubernetes.io/projected/a989c545-d91d-4ca2-9dea-0637a462d204-kube-api-access-kjtq2\") pod \"network-metrics-daemon-bfhh2\" (UID: \"a989c545-d91d-4ca2-9dea-0637a462d204\") " pod="openshift-multus/network-metrics-daemon-bfhh2"
Oct 31 19:20:07.782769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.782737    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnnf\" (UniqueName: \"kubernetes.io/projected/d2d2e4e2-5258-4be1-929f-1d92b689c0ff-kube-api-access-drnnf\") pod \"dns-default-vxmdl\" (UID: \"d2d2e4e2-5258-4be1-929f-1d92b689c0ff\") " pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:07.782856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.782795    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n66g\" (UniqueName: \"kubernetes.io/projected/f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2-kube-api-access-5n66g\") pod \"prometheus-operator-9cd89b74c-t5chn\" (UID: \"f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-t5chn"
Oct 31 19:20:07.782856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.782807    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgx2q\" (UniqueName: \"kubernetes.io/projected/d92bf693-c264-4e52-84a5-045804009257-kube-api-access-bgx2q\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.783033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.783015    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d8m\" (UniqueName: \"kubernetes.io/projected/2825d9bc-c104-43b0-bbe5-acbfd127f71c-kube-api-access-84d8m\") pod \"network-check-target-9xhcv\" (UID: \"2825d9bc-c104-43b0-bbe5-acbfd127f71c\") " pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:07.783210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.783184    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkmw9\" (UniqueName: \"kubernetes.io/projected/a237adff-6089-4d22-993f-95a9869a9771-kube-api-access-nkmw9\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.787169 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.787131    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7sgm\" (UniqueName: \"kubernetes.io/projected/d48cdb47-67b2-47ec-9433-d831d8393f53-kube-api-access-w7sgm\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.787574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.787535    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9qj\" (UniqueName: \"kubernetes.io/projected/31c31f22-7c5d-43bb-b554-ad463e4702c5-kube-api-access-gc9qj\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.787919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.787900    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/162fd1bc-c40c-4001-8fa3-8258ed577b45-kube-api-access-bww5n\") pod \"openshift-state-metrics-7bd7489957-2zxqw\" (UID: \"162fd1bc-c40c-4001-8fa3-8258ed577b45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2zxqw"
Oct 31 19:20:07.787981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.787963    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") pod \"alertmanager-main-0\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:07.788111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.788095    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvnd\" (UniqueName: \"kubernetes.io/projected/f300c7e3-5d90-4bc7-bd34-9d8e743d9f27-kube-api-access-dkvnd\") pod \"kube-state-metrics-5dc8d7d89b-pnwjl\" (UID: \"f300c7e3-5d90-4bc7-bd34-9d8e743d9f27\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-pnwjl"
Oct 31 19:20:07.788928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.788897    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") pod \"console-6c7bffb8dc-997rs\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") " pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:07.861126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861016    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861053    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31c31f22-7c5d-43bb-b554-ad463e4702c5-crio-socket\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.861126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861077    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-etc-kubernetes\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.861126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861105    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-var-lib-kubelet\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861126    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-modprobe-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861133    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861151    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-systemd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861157    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31c31f22-7c5d-43bb-b554-ad463e4702c5-crio-socket\") pod \"insights-runtime-extractor-nx6db\" (UID: \"31c31f22-7c5d-43bb-b554-ad463e4702c5\") " pod="openshift-insights/insights-runtime-extractor-nx6db"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861174    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861199    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861209    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-etc-kubernetes\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861220    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-var-lib-kubelet\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861224    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-bin\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861238    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861252    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-systemd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861264    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-bin\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861268    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-device-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861283    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861308    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-device-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861308    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861332    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-modprobe-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861346    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861350    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-lib-modules\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861385    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-log-socket\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861406    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861437    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-conf-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861449    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-lib-modules\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861463    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-kubernetes\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861484    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-log-socket\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.861493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861503    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-kubernetes\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861524    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-conf-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-sys-fs\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861536    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-d\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861588    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysconfig\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861601    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-sys-fs\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861616    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-host\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861639    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-var-lib-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861658    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-wtmp\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861672    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-sys\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861672    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-host\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861615    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysconfig\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861689    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-multus-certs\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861715    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-var-lib-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861726    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-sys\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861731    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-multus-certs\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861735    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-node-exporter-wtmp\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861729    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-socket-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861770    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-socket-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861798    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-socket-dir-parent\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861813    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-root\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861820    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861828    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-ovn\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861842    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-cnibin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861858    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0312583f-6efa-4d8f-8921-d9deb835d69f-host-slash\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861875    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-root\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861883    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0312583f-6efa-4d8f-8921-d9deb835d69f-host-slash\") pod \"iptables-alerter-9wm8m\" (UID: \"0312583f-6efa-4d8f-8921-d9deb835d69f\") " pod="openshift-network-operator/iptables-alerter-9wm8m"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861901    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-multus-socket-dir-parent\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861896    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-run-ovn\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861916    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-systemd-units\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861919    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-cnibin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861953    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-netns\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861971    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-systemd-units\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861978    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-node-log\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861988    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-run-netns\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.861997    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-node-log\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862003    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-k8s-cni-cncf-io\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862028    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-multus\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862047    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-k8s-cni-cncf-io\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862052    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-dbus\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862078    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-multus\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862084    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a237adff-6089-4d22-993f-95a9869a9771-host\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862095    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-dbus\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862108    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a237adff-6089-4d22-993f-95a9869a9771-host\") pod \"node-ca-6qz8h\" (UID: \"a237adff-6089-4d22-993f-95a9869a9771\") " pod="openshift-image-registry/node-ca-6qz8h"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862109    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-run\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862142    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-kubelet-config\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862169    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-sys\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862192    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d92bf693-c264-4e52-84a5-045804009257-hosts-file\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862198    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/555bb1b5-4eea-4de4-b4a0-45613823a90b-sys\") pod \"node-exporter-l2kfm\" (UID: \"555bb1b5-4eea-4de4-b4a0-45613823a90b\") " pod="openshift-monitoring/node-exporter-l2kfm"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862204    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56c7eb65-3951-480e-88e2-1858cae684e6-kubelet-config\") pod \"global-pull-secret-syncer-bz76x\" (UID: \"56c7eb65-3951-480e-88e2-1858cae684e6\") " pod="kube-system/global-pull-secret-syncer-bz76x"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862168    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-run\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862218    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-etc-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862244    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-netd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862273    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-systemd\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862245    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d92bf693-c264-4e52-84a5-045804009257-hosts-file\") pod \"node-resolver-w6g8x\" (UID: \"d92bf693-c264-4e52-84a5-045804009257\") " pod="openshift-dns/node-resolver-w6g8x"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862307    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-slash\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862332    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-etc-openvswitch\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862338    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-slash\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862335    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-hostroot\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862307    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-cni-netd\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862360    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-systemd\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862367    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-hostroot\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862372    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-kubelet\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862408    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-kubelet\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862447    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862477    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-registration-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862502    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-kubelet\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862529    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862530    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-conf\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862535    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58e5ab86-50ff-419a-a4da-2c19408eb9ae-registration-dir\") pod \"aws-ebs-csi-driver-node-tlbj2\" (UID: \"58e5ab86-50ff-419a-a4da-2c19408eb9ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tlbj2"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862588    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9647c81-f8aa-470d-a5c7-02897a93fa31-host-kubelet\") pod \"ovnkube-node-9njm5\" (UID: \"c9647c81-f8aa-470d-a5c7-02897a93fa31\") " pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-system-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862623    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-netns\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862640    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-bin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862621    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-system-cni-dir\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862684    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-run-netns\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862686    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d48cdb47-67b2-47ec-9433-d831d8393f53-etc-sysctl-conf\") pod \"tuned-7w6mv\" (UID: \"d48cdb47-67b2-47ec-9433-d831d8393f53\") " pod="openshift-cluster-node-tuning-operator/tuned-7w6mv"
Oct 31 19:20:07.862716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:07.862699    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22143d8e-6fcb-4563-8989-a29ddefbc053-host-var-lib-cni-bin\") pod \"multus-jv8rs\" (UID: \"22143d8e-6fcb-4563-8989-a29ddefbc053\") " pod="openshift-multus/multus-jv8rs"
Oct 31 19:20:08.457744 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:08.457701    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:08.812705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:08.812617    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vxmdl"
Oct 31 19:20:08.826533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:08.826505    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b4c88497b-g2wmt"
Oct 31 19:20:11.263488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:11.263450    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:11.281897 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:11.281870    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:11.305621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:11.305589    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:13.464013 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:13.463979    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:15.085673 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:15.085634    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:16.559938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:16.559866    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:16.560291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:16.560097    5958 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:20:16.939634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:16.939605    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79fdb78f65-nscml"]
Oct 31 19:20:17.043178 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:17.043140    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:20:18.758605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:18.758579    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9njm5"
Oct 31 19:20:20.518755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:20.518713    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:21.913856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.913825    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:20:21.914291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.914147    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="alertmanager" containerID="cri-o://89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a" gracePeriod=120
Oct 31 19:20:21.914291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.914205    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-metric" containerID="cri-o://d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43" gracePeriod=120
Oct 31 19:20:21.914291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.914241    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy" containerID="cri-o://7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c" gracePeriod=120
Oct 31 19:20:21.914482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.914258    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="prom-label-proxy" containerID="cri-o://f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7" gracePeriod=120
Oct 31 19:20:21.914482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.914282    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="config-reloader" containerID="cri-o://9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26" gracePeriod=120
Oct 31 19:20:21.914482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:21.914241    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-web" containerID="cri-o://3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d" gracePeriod=120
Oct 31 19:20:22.811278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811243    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7" exitCode=0
Oct 31 19:20:22.811278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811269    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43" exitCode=0
Oct 31 19:20:22.811278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811275    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c" exitCode=0
Oct 31 19:20:22.811278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811283    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26" exitCode=0
Oct 31 19:20:22.811558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811290    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a" exitCode=0
Oct 31 19:20:22.811558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811319    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7"}
Oct 31 19:20:22.811558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811379    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43"}
Oct 31 19:20:22.811558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811404    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c"}
Oct 31 19:20:22.811558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811428    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26"}
Oct 31 19:20:22.811558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:22.811452    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a"}
Oct 31 19:20:23.183141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.183105    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:23.309130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309102    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309134    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309165    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309184    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309229    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309251    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309285    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309339    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309368    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309412    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309453    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309489    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309527    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") pod \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\" (UID: \"0e67bf99-5b7c-4a4d-9b8d-097683e961fb\") "
Oct 31 19:20:23.309782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309644    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:23.309782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309683    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:23.310084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309904    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:23.310084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309936    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-metrics-client-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.310084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.309960    5958 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-main-db\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.312614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.312570    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.313062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.312930    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.313062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.312932    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "kube-api-access-xg6f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:23.313230 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.313043    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:23.313377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.313324    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.313377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.313339    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:23.313478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.313405    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.313730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.313709    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.318146 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.318125    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.324911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.324890    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config" pod "0e67bf99-5b7c-4a4d-9b8d-097683e961fb" (UID: "0e67bf99-5b7c-4a4d-9b8d-097683e961fb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:23.410411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410377    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410411    5958 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410421    5958 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-cluster-tls-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410432    5958 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410466    5958 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-web-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410475    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6f2\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-kube-api-access-xg6f2\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410485    5958 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-config-out\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410494    5958 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-tls-assets\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410503    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-main-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410513    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.410630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.410522    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e67bf99-5b7c-4a4d-9b8d-097683e961fb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:23.818776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.818697    5958 generic.go:356] "Generic (PLEG): container finished" podID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerID="3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d" exitCode=0
Oct 31 19:20:23.818910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.818774    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:23.818910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.818782    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d"}
Oct 31 19:20:23.818910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.818877    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e67bf99-5b7c-4a4d-9b8d-097683e961fb","Type":"ContainerDied","Data":"b27115ecaf8d3f5d89bf549035816ae7e81793c04b1fce5f3c1a115a850c04bb"}
Oct 31 19:20:23.819010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.818945    5958 scope.go:122] "RemoveContainer" containerID="f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7"
Oct 31 19:20:23.828222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.828205    5958 scope.go:122] "RemoveContainer" containerID="d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43"
Oct 31 19:20:23.844587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.844568    5958 scope.go:122] "RemoveContainer" containerID="7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c"
Oct 31 19:20:23.845462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.845410    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:20:23.855609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.855580    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:20:23.856674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.856647    5958 scope.go:122] "RemoveContainer" containerID="3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d"
Oct 31 19:20:23.863930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.863913    5958 scope.go:122] "RemoveContainer" containerID="9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26"
Oct 31 19:20:23.871230 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.871211    5958 scope.go:122] "RemoveContainer" containerID="89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a"
Oct 31 19:20:23.878929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.878911    5958 scope.go:122] "RemoveContainer" containerID="7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6"
Oct 31 19:20:23.885435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885410    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:20:23.885874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885853    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="prom-label-proxy"
Oct 31 19:20:23.885921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885880    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="prom-label-proxy"
Oct 31 19:20:23.885921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885897    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="config-reloader"
Oct 31 19:20:23.885921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885905    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="config-reloader"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885924    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885934    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885944    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="alertmanager"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885952    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="alertmanager"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885969    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-metric"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885977    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-metric"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885992    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="init-config-reloader"
Oct 31 19:20:23.886015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.885999    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="init-config-reloader"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886026    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-web"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886034    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-web"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886113    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886125    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="alertmanager"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886137    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-metric"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886148    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="init-config-reloader"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886160    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="kube-rbac-proxy-web"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886173    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="prom-label-proxy"
Oct 31 19:20:23.886241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886182    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" containerName="config-reloader"
Oct 31 19:20:23.886908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.886877    5958 scope.go:122] "RemoveContainer" containerID="f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7"
Oct 31 19:20:23.887194 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.887176    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7\": container with ID starting with f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7 not found: ID does not exist" containerID="f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7"
Oct 31 19:20:23.887238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.887203    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7"} err="failed to get container status \"f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7\": rpc error: code = NotFound desc = could not find container \"f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7\": container with ID starting with f3b110704d6cfaded6fa821d7051fb9d53514e1b18ed03800d7b93a76d24e3d7 not found: ID does not exist"
Oct 31 19:20:23.887281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.887238    5958 scope.go:122] "RemoveContainer" containerID="d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43"
Oct 31 19:20:23.887514 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.887492    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43\": container with ID starting with d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43 not found: ID does not exist" containerID="d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43"
Oct 31 19:20:23.887616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.887522    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43"} err="failed to get container status \"d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43\": rpc error: code = NotFound desc = could not find container \"d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43\": container with ID starting with d99f38d8f801524f504611465aa5309b1228d6a827301bd3377b9e61251abf43 not found: ID does not exist"
Oct 31 19:20:23.887616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.887554    5958 scope.go:122] "RemoveContainer" containerID="7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c"
Oct 31 19:20:23.887858 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.887837    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c\": container with ID starting with 7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c not found: ID does not exist" containerID="7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c"
Oct 31 19:20:23.887938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.887861    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c"} err="failed to get container status \"7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c\": rpc error: code = NotFound desc = could not find container \"7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c\": container with ID starting with 7fe10c1d4b069caeb289e253d6e432737fcfb239c379914f6caffa1cd36c490c not found: ID does not exist"
Oct 31 19:20:23.887938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.887875    5958 scope.go:122] "RemoveContainer" containerID="3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d"
Oct 31 19:20:23.888113 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.888098    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d\": container with ID starting with 3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d not found: ID does not exist" containerID="3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d"
Oct 31 19:20:23.888152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888125    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d"} err="failed to get container status \"3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d\": rpc error: code = NotFound desc = could not find container \"3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d\": container with ID starting with 3812f9f55052616a12df98d23d8d35659f72fadb2cc27ebd5716f3f5c24afe0d not found: ID does not exist"
Oct 31 19:20:23.888152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888138    5958 scope.go:122] "RemoveContainer" containerID="9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26"
Oct 31 19:20:23.888395 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.888378    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26\": container with ID starting with 9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26 not found: ID does not exist" containerID="9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26"
Oct 31 19:20:23.888443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888401    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26"} err="failed to get container status \"9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26\": rpc error: code = NotFound desc = could not find container \"9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26\": container with ID starting with 9f9df84a12d40713c8f8d40a12f3eeb10075afa15c3acb773da9d20714326e26 not found: ID does not exist"
Oct 31 19:20:23.888443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888415    5958 scope.go:122] "RemoveContainer" containerID="89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a"
Oct 31 19:20:23.888686 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.888669    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a\": container with ID starting with 89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a not found: ID does not exist" containerID="89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a"
Oct 31 19:20:23.888739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888690    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a"} err="failed to get container status \"89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a\": rpc error: code = NotFound desc = could not find container \"89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a\": container with ID starting with 89532671cc2f7ed61be6664fa50e1412dfd2a54896da8f82d08616706230cf7a not found: ID does not exist"
Oct 31 19:20:23.888739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888703    5958 scope.go:122] "RemoveContainer" containerID="7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6"
Oct 31 19:20:23.888933 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:23.888911    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6\": container with ID starting with 7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6 not found: ID does not exist" containerID="7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6"
Oct 31 19:20:23.888979 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.888941    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6"} err="failed to get container status \"7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6\": rpc error: code = NotFound desc = could not find container \"7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6\": container with ID starting with 7f166ec6d9a20d9b02e8fcb9b1010b812b0043d2ea82dd7c2c42f09ef3bda9e6 not found: ID does not exist"
Oct 31 19:20:23.920560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.920515    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:20:23.920708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.920654    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:23.928770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.928740    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 19:20:23.928901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.928847    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 19:20:23.928901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.928894    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 19:20:23.928994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.928973    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 19:20:23.928994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.928977    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 19:20:23.929062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.929001    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 19:20:23.929062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.929009    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 19:20:23.929160 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.929089    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nkp5g\""
Oct 31 19:20:23.929214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.929175    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 19:20:23.935005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:23.934982    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 19:20:24.015054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015022    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-web-config\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015063    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015084    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-config-out\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015176    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015220    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015258    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mxh\" (UniqueName: \"kubernetes.io/projected/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-kube-api-access-l9mxh\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015298    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015324    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015358    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015467    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015499    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015582    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.015692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.015611    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116245    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116294    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116335    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mxh\" (UniqueName: \"kubernetes.io/projected/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-kube-api-access-l9mxh\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116363    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116391    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116436    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116464    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116601    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116660    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116698    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.116890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116740    5958 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 19:20:24.117031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116885    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.117031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116749    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-web-config\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.117031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.116964    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.117031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.117000    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-config-out\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.117449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.117415    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.117815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.117789    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.119508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.119481    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-config-out\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.119684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.119622    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.119684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.119643    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.119906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.119887    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-web-config\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.120003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.119983    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.120113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.120092    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.120490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.120464    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.120657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.120634    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.121442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.121421    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-config-volume\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.130624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.130601    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mxh\" (UniqueName: \"kubernetes.io/projected/c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3-kube-api-access-l9mxh\") pod \"alertmanager-main-0\" (UID: \"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.312648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.312613    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 19:20:24.619477 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.619099    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 19:20:24.621829 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:24.621790    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b8b8d7_f4d1_4fc4_aa0d_49780b322ee3.slice/crio-344df445853abdc679153490333183f035f4ffa9ad29cc0fbd8f6e8696eb47b5 WatchSource:0}: Error finding container 344df445853abdc679153490333183f035f4ffa9ad29cc0fbd8f6e8696eb47b5: Status 404 returned error can't find the container with id 344df445853abdc679153490333183f035f4ffa9ad29cc0fbd8f6e8696eb47b5
Oct 31 19:20:24.661077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.661047    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e67bf99-5b7c-4a4d-9b8d-097683e961fb" path="/var/lib/kubelet/pods/0e67bf99-5b7c-4a4d-9b8d-097683e961fb/volumes"
Oct 31 19:20:24.825237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.825197    5958 generic.go:356] "Generic (PLEG): container finished" podID="c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3" containerID="16b456d8cfd13eeeb62d18dcdde11a84cc36a8983d445406393211f3ae999c7d" exitCode=0
Oct 31 19:20:24.825392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.825265    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerDied","Data":"16b456d8cfd13eeeb62d18dcdde11a84cc36a8983d445406393211f3ae999c7d"}
Oct 31 19:20:24.825392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:24.825331    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"344df445853abdc679153490333183f035f4ffa9ad29cc0fbd8f6e8696eb47b5"}
Oct 31 19:20:25.837114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.835005    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"38c2bbbe1a064728f21ec372112648f57f831fd416d0888b547ae68b0eb90107"}
Oct 31 19:20:25.837114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.835086    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"65c7ceb0274ef18c7580fd47fb4054e0178f570523026b5a424d2980a6ac5325"}
Oct 31 19:20:25.837114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.835128    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"90ab3d9880ae7fff203f055a7d400c15292c56ae779d79fe385904fb6d1e9a80"}
Oct 31 19:20:25.837114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.835166    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"6d5e0594d3c4a50acb09d57172abf74530eeea8a65996dc4e0e5d5da7aaa310a"}
Oct 31 19:20:25.837114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.835202    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"cde46db49bcaf3e239e823f98ad3e5f9dcf2644f60fd1bec63b5e16041deb9a7"}
Oct 31 19:20:25.837114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.835241    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3","Type":"ContainerStarted","Data":"c978171e072d4edf74fd41d00575b9052ad01a1ba5f873757b881148be7fdf6f"}
Oct 31 19:20:25.919457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.919385    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9064006449999997 podStartE2EDuration="2.919361536s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.960891ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:25.87093378 +0000 UTC m=+19.332308404" watchObservedRunningTime="2026-10-31 19:20:25.919361536 +0000 UTC m=+19.380736151"
Oct 31 19:20:25.919773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.919754    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"]
Oct 31 19:20:25.938023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.937994    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"]
Oct 31 19:20:25.938184 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.938120    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:25.943518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.943495    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 19:20:25.946405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.946378    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 19:20:25.992954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.992920    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 19:20:25.993131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.992980    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 19:20:25.993131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.992983    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 19:20:25.993235 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.993172    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ff25q\""
Oct 31 19:20:25.993235 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:25.993183    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 19:20:26.035850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.035816    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-secret-telemeter-client\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.035945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.035860    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-metrics-client-ca\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.035945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.035939    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-federate-client-tls\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.036043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.036001    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.036043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.036023    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5vwm\" (UniqueName: \"kubernetes.io/projected/ec5f7e45-b28c-457d-afd8-92caec86aeb8-kube-api-access-r5vwm\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.036147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.036073    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.036147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.036113    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-telemeter-client-tls\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.036147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.036136    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.136903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.136866    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-secret-telemeter-client\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.136913    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-metrics-client-ca\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.136972    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-federate-client-tls\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.137015    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.137040    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5vwm\" (UniqueName: \"kubernetes.io/projected/ec5f7e45-b28c-457d-afd8-92caec86aeb8-kube-api-access-r5vwm\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.137075    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.137123    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-telemeter-client-tls\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.137403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.137156    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.139644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.139614    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.140023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.139994    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-metrics-client-ca\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.140151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.140086    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5f7e45-b28c-457d-afd8-92caec86aeb8-serving-certs-ca-bundle\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.141010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.140982    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.141817 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.141793    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-telemeter-client-tls\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.141929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.141913    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-secret-telemeter-client\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.141998 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.141941    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ec5f7e45-b28c-457d-afd8-92caec86aeb8-federate-client-tls\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.153243 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.153215    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5vwm\" (UniqueName: \"kubernetes.io/projected/ec5f7e45-b28c-457d-afd8-92caec86aeb8-kube-api-access-r5vwm\") pod \"telemeter-client-7f9b87db9-jxwtc\" (UID: \"ec5f7e45-b28c-457d-afd8-92caec86aeb8\") " pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.317076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.317040    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359087    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359399    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="prometheus" containerID="cri-o://d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f" gracePeriod=600
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359645    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy" containerID="cri-o://7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a" gracePeriod=600
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359701    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="thanos-sidecar" containerID="cri-o://4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d" gracePeriod=600
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359714    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="config-reloader" containerID="cri-o://bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492" gracePeriod=600
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359780    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-web" containerID="cri-o://10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b" gracePeriod=600
Oct 31 19:20:26.361863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.359655    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1" gracePeriod=600
Oct 31 19:20:26.385249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.385221    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:26.653861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.653828    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f9b87db9-jxwtc"]
Oct 31 19:20:26.657376 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:26.657346    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5f7e45_b28c_457d_afd8_92caec86aeb8.slice/crio-cfa8e2c9144741f5948c4dcb32b7f6dd46f59b586fcbdd2ecf30f1e89c179507 WatchSource:0}: Error finding container cfa8e2c9144741f5948c4dcb32b7f6dd46f59b586fcbdd2ecf30f1e89c179507: Status 404 returned error can't find the container with id cfa8e2c9144741f5948c4dcb32b7f6dd46f59b586fcbdd2ecf30f1e89c179507
Oct 31 19:20:26.662027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.659751    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:20:26.698636 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.698597    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:26.839653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.839613    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc" event={"ID":"ec5f7e45-b28c-457d-afd8-92caec86aeb8","Type":"ContainerStarted","Data":"cfa8e2c9144741f5948c4dcb32b7f6dd46f59b586fcbdd2ecf30f1e89c179507"}
Oct 31 19:20:26.847393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847358    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1" exitCode=0
Oct 31 19:20:26.847393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847381    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a" exitCode=0
Oct 31 19:20:26.847393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847387    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b" exitCode=0
Oct 31 19:20:26.847393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847395    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d" exitCode=0
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847402    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492" exitCode=0
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847411    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f" exitCode=0
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847416    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"}
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847463    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847531    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"}
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847599    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.847693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847590    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"}
Oct 31 19:20:26.847928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847715    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"}
Oct 31 19:20:26.847928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847757    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"}
Oct 31 19:20:26.847928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847796    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"}
Oct 31 19:20:26.847928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.847833    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdcc1f0-1892-4f50-9b50-2c6b864b502e","Type":"ContainerDied","Data":"b36404272e752f9901f29245d0dac0d8631191537e680927a25651ffacfa79ba"}
Oct 31 19:20:26.850774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.850757    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.850866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.850793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.850866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.850813    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.850975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.850940    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.850977    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851004    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851033    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851061    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851117    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851144    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851165    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.851320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851173    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851226    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.851320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851240    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851294    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851459    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851498    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851534    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851578    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851596    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851604    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.851621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851608    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.851921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851637    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.851921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.851683    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\" (UID: \"8bdcc1f0-1892-4f50-9b50-2c6b864b502e\") "
Oct 31 19:20:26.852413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852136    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.852413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852154    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.852413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852161    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.852413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852201    5958 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-metrics-client-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.852413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852219    5958 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.852413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852236    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.852775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.852661    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:26.854205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.854178    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "kube-api-access-wbg2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:26.854417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.854391    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.854528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.854503    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.854903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.854871    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:26.854903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.854990    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.855488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.855467    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:26.856048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.856015    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:26.856280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.856253    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.856767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.856741    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.857154 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.857134    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.857343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.857323    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.857447 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.857432    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.858716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.858700    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.866821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.866801    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.869242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.869217    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config" pod "8bdcc1f0-1892-4f50-9b50-2c6b864b502e" (UID: "8bdcc1f0-1892-4f50-9b50-2c6b864b502e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:26.875044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.875027    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.882578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.882535    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.891481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.891459    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.899501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.899474    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.907149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.907125    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.907470 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.907454    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.907518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.907481    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"} err="failed to get container status \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist"
Oct 31 19:20:26.907518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.907500    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.907772 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.907755    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.907823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.907777    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"} err="failed to get container status \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist"
Oct 31 19:20:26.907823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.907792    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.908021 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.908005    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.908075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908025    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"} err="failed to get container status \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist"
Oct 31 19:20:26.908075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908039    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.908281 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.908262    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.908324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908287    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"} err="failed to get container status \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist"
Oct 31 19:20:26.908324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908303    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.908613 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.908589    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.908670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908620    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"} err="failed to get container status \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist"
Oct 31 19:20:26.908670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908642    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.908897 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.908877    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.908933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908911    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"} err="failed to get container status \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist"
Oct 31 19:20:26.908933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.908926    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.909193 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:26.909178    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.909236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909198    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"} err="failed to get container status \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist"
Oct 31 19:20:26.909236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909210    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.909426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909409    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"} err="failed to get container status \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist"
Oct 31 19:20:26.909465 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909427    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.909675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909652    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"} err="failed to get container status \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist"
Oct 31 19:20:26.909675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909673    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.909955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909928    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"} err="failed to get container status \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist"
Oct 31 19:20:26.909992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.909956    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.910194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910179    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"} err="failed to get container status \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist"
Oct 31 19:20:26.910288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910195    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.910437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910417    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"} err="failed to get container status \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist"
Oct 31 19:20:26.910472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910439    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.910766 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910748    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"} err="failed to get container status \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist"
Oct 31 19:20:26.910809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910767    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.910987 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910972    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"} err="failed to get container status \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist"
Oct 31 19:20:26.911038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.910987    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.911193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911171    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"} err="failed to get container status \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist"
Oct 31 19:20:26.911193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911191    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.911432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911415    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"} err="failed to get container status \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist"
Oct 31 19:20:26.911432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911431    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.911652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911635    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"} err="failed to get container status \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist"
Oct 31 19:20:26.911652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911652    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.911879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911862    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"} err="failed to get container status \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist"
Oct 31 19:20:26.911921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.911879    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.912106 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912087    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"} err="failed to get container status \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist"
Oct 31 19:20:26.912147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912107    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.912313 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912295    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"} err="failed to get container status \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist"
Oct 31 19:20:26.912355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912314    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.912537 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912520    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"} err="failed to get container status \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist"
Oct 31 19:20:26.912642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912538    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.912784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912768    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"} err="failed to get container status \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist"
Oct 31 19:20:26.912828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912784    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.913005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.912988    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"} err="failed to get container status \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist"
Oct 31 19:20:26.913043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913006    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.913220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913203    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"} err="failed to get container status \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist"
Oct 31 19:20:26.913257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913220    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.913437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913419    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"} err="failed to get container status \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist"
Oct 31 19:20:26.913478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913438    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.913659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913642    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"} err="failed to get container status \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist"
Oct 31 19:20:26.913705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913659    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.913860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913845    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"} err="failed to get container status \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist"
Oct 31 19:20:26.913903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.913862    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.914099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914080    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"} err="failed to get container status \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist"
Oct 31 19:20:26.914138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914100    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.914303 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914287    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"} err="failed to get container status \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist"
Oct 31 19:20:26.914351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914303    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.914530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914513    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"} err="failed to get container status \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist"
Oct 31 19:20:26.914530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914530    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.914758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914742    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"} err="failed to get container status \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist"
Oct 31 19:20:26.914758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914757    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.914966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914948    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"} err="failed to get container status \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist"
Oct 31 19:20:26.915014 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.914967    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.915192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915176    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"} err="failed to get container status \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist"
Oct 31 19:20:26.915233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915193    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.915396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915380    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"} err="failed to get container status \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist"
Oct 31 19:20:26.915434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915397    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.915597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915582    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"} err="failed to get container status \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist"
Oct 31 19:20:26.915641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915596    5958 scope.go:122] "RemoveContainer" containerID="28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"
Oct 31 19:20:26.915800 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915783    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1"} err="failed to get container status \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": rpc error: code = NotFound desc = could not find container \"28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1\": container with ID starting with 28a6dbaced78343cdbe4668c58eeaee1e8dedbbf24b6e5acc6c4c2ee8d3920d1 not found: ID does not exist"
Oct 31 19:20:26.915842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915802    5958 scope.go:122] "RemoveContainer" containerID="7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"
Oct 31 19:20:26.916005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.915990    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a"} err="failed to get container status \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": rpc error: code = NotFound desc = could not find container \"7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a\": container with ID starting with 7de540d5236637aff09f5ac6a838af5dc1f5a6d65cc101c10bce0ecc00d2101a not found: ID does not exist"
Oct 31 19:20:26.916005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916004    5958 scope.go:122] "RemoveContainer" containerID="10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"
Oct 31 19:20:26.916216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916197    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b"} err="failed to get container status \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": rpc error: code = NotFound desc = could not find container \"10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b\": container with ID starting with 10db7b57cf1e001c74b834fc0b38b1500fdb85ad90aeff32827a054b2e581d7b not found: ID does not exist"
Oct 31 19:20:26.916265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916216    5958 scope.go:122] "RemoveContainer" containerID="4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"
Oct 31 19:20:26.916416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916401    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d"} err="failed to get container status \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": rpc error: code = NotFound desc = could not find container \"4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d\": container with ID starting with 4f6a4b938f02ba9e10d4efe63a82a9cce5340b1037d87638578cb9974045459d not found: ID does not exist"
Oct 31 19:20:26.916416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916415    5958 scope.go:122] "RemoveContainer" containerID="bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"
Oct 31 19:20:26.916645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916625    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492"} err="failed to get container status \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": rpc error: code = NotFound desc = could not find container \"bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492\": container with ID starting with bc5c860062cd838e0fb0fe58fdc169a31a6fd5a7bd1da66a5c90068229b06492 not found: ID does not exist"
Oct 31 19:20:26.916645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916645    5958 scope.go:122] "RemoveContainer" containerID="d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"
Oct 31 19:20:26.916849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916826    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f"} err="failed to get container status \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": rpc error: code = NotFound desc = could not find container \"d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f\": container with ID starting with d409e0279d5caf1f09c2f4e4505cfbad473a6c8ebb0f0b24b551e67f37c34c4f not found: ID does not exist"
Oct 31 19:20:26.916887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.916849    5958 scope.go:122] "RemoveContainer" containerID="9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"
Oct 31 19:20:26.917047 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.917030    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e"} err="failed to get container status \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": rpc error: code = NotFound desc = could not find container \"9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e\": container with ID starting with 9ca200b34ad31e94caaf4aba6b1098a9aef7c2faa9ad38a09290c23a4bb8007e not found: ID does not exist"
Oct 31 19:20:26.953777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953741    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.953777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953778    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbg2t\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-kube-api-access-wbg2t\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.953777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953799    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-prometheus-k8s-db\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953814    5958 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-grpc-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953829    5958 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-kube-rbac-proxy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953846    5958 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config-out\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953861    5958 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-metrics-client-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953876    5958 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-tls-assets\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953891    5958 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953905    5958 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-web-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953922    5958 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953938    5958 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953953    5958 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953971    5958 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:26.954046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:26.953988    5958 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdcc1f0-1892-4f50-9b50-2c6b864b502e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:27.179253 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.179213    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:20:27.184089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.184062    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:20:27.215896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.215859    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:20:27.216226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216211    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="prometheus"
Oct 31 19:20:27.216274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216228    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="prometheus"
Oct 31 19:20:27.216274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216238    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="init-config-reloader"
Oct 31 19:20:27.216274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216244    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="init-config-reloader"
Oct 31 19:20:27.216274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216252    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-thanos"
Oct 31 19:20:27.216274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216261    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-thanos"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216279    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216285    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216292    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="config-reloader"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216296    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="config-reloader"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216303    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-web"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216308    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-web"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216320    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="thanos-sidecar"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216325    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="thanos-sidecar"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216378    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="prometheus"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216388    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="init-config-reloader"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216393    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-thanos"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216400    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216405    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="config-reloader"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216410    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="kube-rbac-proxy-web"
Oct 31 19:20:27.216434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.216417    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" containerName="thanos-sidecar"
Oct 31 19:20:27.250409 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.250380    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:20:27.250589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.250517    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.256007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.255964    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 19:20:27.256178 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.255975    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 19:20:27.256359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.256335    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 19:20:27.258985 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.258964    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 19:20:27.259111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259051    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 19:20:27.259111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259069    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 19:20:27.259222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259110    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 19:20:27.259222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259146    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 19:20:27.259222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259188    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 19:20:27.259367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259323    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 19:20:27.259367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259333    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dbvnt\""
Oct 31 19:20:27.259367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259346    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 19:20:27.259485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259413    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4405tprkifkut\""
Oct 31 19:20:27.259521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259503    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 19:20:27.259675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.259660    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 19:20:27.261939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.261923    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 19:20:27.357451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357411    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357515    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357564    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357594    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357637    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357701    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357755    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e212a43f-dd93-4653-ae12-87381b6fbb97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357774    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357790    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357811    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357861    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-web-config\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.357946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357942    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e212a43f-dd93-4653-ae12-87381b6fbb97-config-out\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357960    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.357988    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.358027    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.358058    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.358089    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-config\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.358112    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.358138    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.358280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.358163    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmx7\" (UniqueName: \"kubernetes.io/projected/e212a43f-dd93-4653-ae12-87381b6fbb97-kube-api-access-rbmx7\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.458749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.458658    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.458749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.458708    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.458984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.458884    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.458984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.458940    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.458984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.458972    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459026    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e212a43f-dd93-4653-ae12-87381b6fbb97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459052    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459077    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459113    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459146    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-web-config\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459187    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e212a43f-dd93-4653-ae12-87381b6fbb97-config-out\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459209    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459240    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459283    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459313    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459342    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-config\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459368    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459395    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459420    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmx7\" (UniqueName: \"kubernetes.io/projected/e212a43f-dd93-4653-ae12-87381b6fbb97-kube-api-access-rbmx7\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459426    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.459774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.459463    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.460794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.460760    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.461237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.461213    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.461676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.461651    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.462075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.462046    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.462361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.462335    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.462619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.462594    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.462921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.462894    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e212a43f-dd93-4653-ae12-87381b6fbb97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.463332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.463307    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.463700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.463671    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.464782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.464761    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.464879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.464769    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.465183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.465089    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.465272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.465118    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e212a43f-dd93-4653-ae12-87381b6fbb97-config-out\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.465826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.465701    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e212a43f-dd93-4653-ae12-87381b6fbb97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.466033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.466013    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-web-config\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.466871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.466846    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.466965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.466950    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-config\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.467497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.467477    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e212a43f-dd93-4653-ae12-87381b6fbb97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.474276 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.474248    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmx7\" (UniqueName: \"kubernetes.io/projected/e212a43f-dd93-4653-ae12-87381b6fbb97-kube-api-access-rbmx7\") pod \"prometheus-k8s-0\" (UID: \"e212a43f-dd93-4653-ae12-87381b6fbb97\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:27.698272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:27.698230    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:28.021075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.021040    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 19:20:28.023028 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:20:28.023001    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode212a43f_dd93_4653_ae12_87381b6fbb97.slice/crio-cd53e56f6e61c4baf70bd137f87276da4495bd278acab6f72922761ea2b0b280 WatchSource:0}: Error finding container cd53e56f6e61c4baf70bd137f87276da4495bd278acab6f72922761ea2b0b280: Status 404 returned error can't find the container with id cd53e56f6e61c4baf70bd137f87276da4495bd278acab6f72922761ea2b0b280
Oct 31 19:20:28.463421 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.463382    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:28.468593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.468565    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-674ffd47d7-5p6sd"
Oct 31 19:20:28.661782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.661753    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdcc1f0-1892-4f50-9b50-2c6b864b502e" path="/var/lib/kubelet/pods/8bdcc1f0-1892-4f50-9b50-2c6b864b502e/volumes"
Oct 31 19:20:28.858137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.858052    5958 generic.go:356] "Generic (PLEG): container finished" podID="e212a43f-dd93-4653-ae12-87381b6fbb97" containerID="e2c3bd285ea03cff592acb2aee9f789a5c2ee35c9eed8c607285344a42436d9b" exitCode=0
Oct 31 19:20:28.858294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.858141    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerDied","Data":"e2c3bd285ea03cff592acb2aee9f789a5c2ee35c9eed8c607285344a42436d9b"}
Oct 31 19:20:28.858294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:28.858234    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"cd53e56f6e61c4baf70bd137f87276da4495bd278acab6f72922761ea2b0b280"}
Oct 31 19:20:29.864977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.864944    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc" event={"ID":"ec5f7e45-b28c-457d-afd8-92caec86aeb8","Type":"ContainerStarted","Data":"6f2de2ef5e00d999160e9740f28c87290e3d3ec8dd04817c6f48903ba327bed4"}
Oct 31 19:20:29.865439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.865043    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc" event={"ID":"ec5f7e45-b28c-457d-afd8-92caec86aeb8","Type":"ContainerStarted","Data":"dded46fdc0cdc07cf75407fb36139785046d6905751919ddd40db8a1cc29f0e4"}
Oct 31 19:20:29.865439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.865066    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc" event={"ID":"ec5f7e45-b28c-457d-afd8-92caec86aeb8","Type":"ContainerStarted","Data":"967c818456be3f3efab84b3c2a68a0890e70eedfab2a98cce35579b10400baed"}
Oct 31 19:20:29.868218 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.868194    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"a6cd5d0df47ea17e9cab9a9c5d8ce6223aae18c7d1d8c4ca34470a706b4577e0"}
Oct 31 19:20:29.868336 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.868253    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"982ac99946e20d6d2e55c624af827feba5d7bd214925a62d09288255bfb2416c"}
Oct 31 19:20:29.868336 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.868284    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"1c572820bfe54cdea28758997a9271ad9ae32313bc8d262a62cf0c9c37aefbf6"}
Oct 31 19:20:29.868336 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.868309    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"fec5ebf22d2e3266c5b071b7d03b962d73798694e528fffc471e1f2246437792"}
Oct 31 19:20:29.868336 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.868331    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"9e7e1cc5ad9b54a5995d5676ffad2e77c475d615302a203ba1a3de085286f28d"}
Oct 31 19:20:29.868481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.868356    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e212a43f-dd93-4653-ae12-87381b6fbb97","Type":"ContainerStarted","Data":"d43780d9eb3334539fa63d41923e065177397a2eb7c5936d5f8ba10fa825e623"}
Oct 31 19:20:29.892471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.892404    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f9b87db9-jxwtc" podStartSLOduration=2.516235398 podStartE2EDuration="4.892387596s" totalImagesPullingTime="2.376152198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:29.891150013 +0000 UTC m=+23.352524653" watchObservedRunningTime="2026-10-31 19:20:29.892387596 +0000 UTC m=+23.353762213"
Oct 31 19:20:29.928245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.928144    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.908593485 podStartE2EDuration="2.928126646s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.533161ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:29.92758004 +0000 UTC m=+23.388954655" watchObservedRunningTime="2026-10-31 19:20:29.928126646 +0000 UTC m=+23.389501263"
Oct 31 19:20:29.949307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:29.949281    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgtp6"
Oct 31 19:20:32.655688 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:32.655640    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c7bffb8dc-997rs" podUID="5d1411a2-289b-443e-ba7c-663c8f931635" containerName="console" containerID="cri-o://1c23d39da7f3fbd630002a27177e6227f12d22e31a31b8cc29fce678056c1e85" gracePeriod=15
Oct 31 19:20:32.698405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:32.698371    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:20:32.881296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:32.881264    5958 generic.go:356] "Generic (PLEG): container finished" podID="5d1411a2-289b-443e-ba7c-663c8f931635" containerID="1c23d39da7f3fbd630002a27177e6227f12d22e31a31b8cc29fce678056c1e85" exitCode=0
Oct 31 19:20:32.881491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:32.881337    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bffb8dc-997rs" event={"ID":"5d1411a2-289b-443e-ba7c-663c8f931635","Type":"ContainerDied","Data":"1c23d39da7f3fbd630002a27177e6227f12d22e31a31b8cc29fce678056c1e85"}
Oct 31 19:20:32.932855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:32.932822    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:33.018398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018353    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.018634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018442    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.018634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018475    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.018634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018501    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.018634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018601    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.018861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018650    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.018861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018664    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:33.018861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018695    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") pod \"5d1411a2-289b-443e-ba7c-663c8f931635\" (UID: \"5d1411a2-289b-443e-ba7c-663c8f931635\") "
Oct 31 19:20:33.019012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.018892    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:33.019072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.019016    5958 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-oauth-serving-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.019072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.019035    5958 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d1411a2-289b-443e-ba7c-663c8f931635-tmp\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.019138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.019093    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:33.019424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.019398    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:33.020981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.020958    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:33.021068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.021044    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:33.021068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.021056    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s" pod "5d1411a2-289b-443e-ba7c-663c8f931635" (UID: "5d1411a2-289b-443e-ba7c-663c8f931635"). InnerVolumeSpecName "kube-api-access-dvf4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:33.119762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.119718    5958 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-oauth-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.119762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.119753    5958 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-console-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.119762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.119764    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvf4s\" (UniqueName: \"kubernetes.io/projected/5d1411a2-289b-443e-ba7c-663c8f931635-kube-api-access-dvf4s\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.119762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.119773    5958 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1411a2-289b-443e-ba7c-663c8f931635-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.120069 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.119783    5958 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d1411a2-289b-443e-ba7c-663c8f931635-console-serving-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:33.886934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.886908    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7bffb8dc-997rs"
Oct 31 19:20:33.887329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.886906    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7bffb8dc-997rs" event={"ID":"5d1411a2-289b-443e-ba7c-663c8f931635","Type":"ContainerDied","Data":"24c8d577dc2f399125ecc9d527acecb8b0e1f20d03c3650c7e0db00d442b9486"}
Oct 31 19:20:33.887329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.887043    5958 scope.go:122] "RemoveContainer" containerID="1c23d39da7f3fbd630002a27177e6227f12d22e31a31b8cc29fce678056c1e85"
Oct 31 19:20:33.913755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.913718    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7bffb8dc-997rs"]
Oct 31 19:20:33.917875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:33.917848    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7bffb8dc-997rs"]
Oct 31 19:20:34.663094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:34.663062    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1411a2-289b-443e-ba7c-663c8f931635" path="/var/lib/kubelet/pods/5d1411a2-289b-443e-ba7c-663c8f931635/volumes"
Oct 31 19:20:35.088846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:35.088754    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9xhcv"
Oct 31 19:20:41.964701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:41.964642    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79fdb78f65-nscml" podUID="55adc602-d71a-4b6c-be3e-5033bd5d6393" containerName="console" containerID="cri-o://25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e" gracePeriod=15
Oct 31 19:20:42.203892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.203865    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:42.304976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.304892    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.304976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.304948    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.304986    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305016    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305064    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305090    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305161    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305202    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") pod \"55adc602-d71a-4b6c-be3e-5033bd5d6393\" (UID: \"55adc602-d71a-4b6c-be3e-5033bd5d6393\") "
Oct 31 19:20:42.305504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305278    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:42.305504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305471    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:42.305504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305479    5958 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55adc602-d71a-4b6c-be3e-5033bd5d6393-tmp\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.305504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305488    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:42.305790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305619    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:42.305790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.305615    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:42.307420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.307388    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:42.307527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.307439    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "kube-api-access-46f7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:42.307527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.307449    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config" pod "55adc602-d71a-4b6c-be3e-5033bd5d6393" (UID: "55adc602-d71a-4b6c-be3e-5033bd5d6393"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:20:42.406523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406482    5958 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.406523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406517    5958 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-oauth-serving-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.406523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406526    5958 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-trusted-ca-bundle\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.406795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406535    5958 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-serving-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.406795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406576    5958 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55adc602-d71a-4b6c-be3e-5033bd5d6393-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.406795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406586    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46f7z\" (UniqueName: \"kubernetes.io/projected/55adc602-d71a-4b6c-be3e-5033bd5d6393-kube-api-access-46f7z\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.406795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.406596    5958 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55adc602-d71a-4b6c-be3e-5033bd5d6393-console-oauth-config\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:42.924927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.924844    5958 generic.go:356] "Generic (PLEG): container finished" podID="55adc602-d71a-4b6c-be3e-5033bd5d6393" containerID="25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e" exitCode=0
Oct 31 19:20:42.924927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.924897    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79fdb78f65-nscml"
Oct 31 19:20:42.925111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.924898    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fdb78f65-nscml" event={"ID":"55adc602-d71a-4b6c-be3e-5033bd5d6393","Type":"ContainerDied","Data":"25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e"}
Oct 31 19:20:42.925111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.925026    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79fdb78f65-nscml" event={"ID":"55adc602-d71a-4b6c-be3e-5033bd5d6393","Type":"ContainerDied","Data":"feaf09c7f9aebe1024f4cad13cf0f164ca47cb0f0988384ff8c8d26a33eab8c6"}
Oct 31 19:20:42.925111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.925053    5958 scope.go:122] "RemoveContainer" containerID="25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e"
Oct 31 19:20:42.933247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.933226    5958 scope.go:122] "RemoveContainer" containerID="25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e"
Oct 31 19:20:42.933586 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:42.933566    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e\": container with ID starting with 25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e not found: ID does not exist" containerID="25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e"
Oct 31 19:20:42.933638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.933597    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e"} err="failed to get container status \"25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e\": rpc error: code = NotFound desc = could not find container \"25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e\": container with ID starting with 25549e034fe28f0ce14b74893baa9d630e8f0cd68fb65cdc7fe97f443b29a68e not found: ID does not exist"
Oct 31 19:20:42.946156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.946126    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79fdb78f65-nscml"]
Oct 31 19:20:42.953971 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:42.953945    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79fdb78f65-nscml"]
Oct 31 19:20:44.661087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:20:44.661052    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55adc602-d71a-4b6c-be3e-5033bd5d6393" path="/var/lib/kubelet/pods/55adc602-d71a-4b6c-be3e-5033bd5d6393/volumes"
Oct 31 19:20:47.043012 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:20:47.042979    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:21:11.751767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:21:11.751721    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:21:11.767819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:21:11.767792    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:21:12.049878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:21:12.049797    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 19:21:17.043236 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:21:17.043195    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:21:47.043976 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:21:47.043885    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:22:17.044193 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:22:17.044151    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:22:47.043353 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:22:47.043309    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:23:17.044585 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:23:17.044487    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:23:19.280683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.280649    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"]
Oct 31 19:23:19.281064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.280988    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5d1411a2-289b-443e-ba7c-663c8f931635" containerName="console"
Oct 31 19:23:19.281064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.280998    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1411a2-289b-443e-ba7c-663c8f931635" containerName="console"
Oct 31 19:23:19.281064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.281035    5958 cpu_manager.go:414] "containerMap: removing container" podUID="55adc602-d71a-4b6c-be3e-5033bd5d6393" containerName="console"
Oct 31 19:23:19.281064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.281040    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55adc602-d71a-4b6c-be3e-5033bd5d6393" containerName="console"
Oct 31 19:23:19.281207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.281156    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="55adc602-d71a-4b6c-be3e-5033bd5d6393" containerName="console"
Oct 31 19:23:19.281207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.281167    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1411a2-289b-443e-ba7c-663c8f931635" containerName="console"
Oct 31 19:23:19.284383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.284365    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.293689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.293665    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 19:23:19.293689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.293665    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 19:23:19.293894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.293881    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-j4zhp\""
Oct 31 19:23:19.307664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.307638    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"]
Oct 31 19:23:19.401183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.401147    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.401354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.401214    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2f5x\" (UniqueName: \"kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.401354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.401314    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.502001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.501961    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.502182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.502030    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2f5x\" (UniqueName: \"kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.502182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.502064    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.502375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.502356    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.502419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.502382    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.516834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.516805    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2f5x\" (UniqueName: \"kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.600321 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.600228    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:19.875187 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:19.875147    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"]
Oct 31 19:23:19.878082 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:23:19.878041    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f58e34e_789b_43c4_b4a3_e4fa843cf943.slice/crio-70a496c650d977639d8a1dd180f84a7001a113cd8a48835d7e535dec779dc157 WatchSource:0}: Error finding container 70a496c650d977639d8a1dd180f84a7001a113cd8a48835d7e535dec779dc157: Status 404 returned error can't find the container with id 70a496c650d977639d8a1dd180f84a7001a113cd8a48835d7e535dec779dc157
Oct 31 19:23:20.544964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:20.544921    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" event={"ID":"6f58e34e-789b-43c4-b4a3-e4fa843cf943","Type":"ContainerStarted","Data":"70a496c650d977639d8a1dd180f84a7001a113cd8a48835d7e535dec779dc157"}
Oct 31 19:23:25.570598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:25.570473    5958 generic.go:356] "Generic (PLEG): container finished" podID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerID="3e9449d84a3ab8bf16b3f0516dcbbf27b858e8f9b0cfbb472834286b77f4253a" exitCode=0
Oct 31 19:23:25.570598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:25.570581    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" event={"ID":"6f58e34e-789b-43c4-b4a3-e4fa843cf943","Type":"ContainerDied","Data":"3e9449d84a3ab8bf16b3f0516dcbbf27b858e8f9b0cfbb472834286b77f4253a"}
Oct 31 19:23:28.585217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:28.585184    5958 generic.go:356] "Generic (PLEG): container finished" podID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerID="099e943a549ca0c250b0d3ac94d20f18190632378f7cf8273803c0cc01fa942f" exitCode=0
Oct 31 19:23:28.585650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:28.585248    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" event={"ID":"6f58e34e-789b-43c4-b4a3-e4fa843cf943","Type":"ContainerDied","Data":"099e943a549ca0c250b0d3ac94d20f18190632378f7cf8273803c0cc01fa942f"}
Oct 31 19:23:34.612284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:34.612249    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" event={"ID":"6f58e34e-789b-43c4-b4a3-e4fa843cf943","Type":"ContainerStarted","Data":"646a772352b0ddfcd099718ed1a9ec6a7a32e56bfeb930e499d273cbf5fd7316"}
Oct 31 19:23:34.639696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:34.639639    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" podStartSLOduration=1.999001878 podStartE2EDuration="15.639621113s" totalImagesPullingTime="13.640619235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:34.638873532 +0000 UTC m=+208.100248184" watchObservedRunningTime="2026-10-31 19:23:34.639621113 +0000 UTC m=+208.100995725"
Oct 31 19:23:35.617972 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:35.617937    5958 generic.go:356] "Generic (PLEG): container finished" podID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerID="646a772352b0ddfcd099718ed1a9ec6a7a32e56bfeb930e499d273cbf5fd7316" exitCode=0
Oct 31 19:23:35.618457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:35.618026    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" event={"ID":"6f58e34e-789b-43c4-b4a3-e4fa843cf943","Type":"ContainerDied","Data":"646a772352b0ddfcd099718ed1a9ec6a7a32e56bfeb930e499d273cbf5fd7316"}
Oct 31 19:23:36.748841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.748817    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:36.864616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.864587    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x\" (UniqueName: \"kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x\") pod \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") "
Oct 31 19:23:36.864823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.864639    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle\") pod \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") "
Oct 31 19:23:36.864823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.864683    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util\") pod \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\" (UID: \"6f58e34e-789b-43c4-b4a3-e4fa843cf943\") "
Oct 31 19:23:36.865418 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.865388    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle" pod "6f58e34e-789b-43c4-b4a3-e4fa843cf943" (UID: "6f58e34e-789b-43c4-b4a3-e4fa843cf943"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:36.867277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.867245    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x" pod "6f58e34e-789b-43c4-b4a3-e4fa843cf943" (UID: "6f58e34e-789b-43c4-b4a3-e4fa843cf943"). InnerVolumeSpecName "kube-api-access-d2f5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:36.869525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.869496    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util" pod "6f58e34e-789b-43c4-b4a3-e4fa843cf943" (UID: "6f58e34e-789b-43c4-b4a3-e4fa843cf943"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:36.965592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.965471    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2f5x\" (UniqueName: \"kubernetes.io/projected/6f58e34e-789b-43c4-b4a3-e4fa843cf943-kube-api-access-d2f5x\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:23:36.965592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.965503    5958 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-bundle\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:23:36.965592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:36.965512    5958 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f58e34e-789b-43c4-b4a3-e4fa843cf943-util\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:23:37.627556 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:37.627529    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr"
Oct 31 19:23:37.627707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:37.627532    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8cmhr" event={"ID":"6f58e34e-789b-43c4-b4a3-e4fa843cf943","Type":"ContainerDied","Data":"70a496c650d977639d8a1dd180f84a7001a113cd8a48835d7e535dec779dc157"}
Oct 31 19:23:37.627707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:37.627629    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a496c650d977639d8a1dd180f84a7001a113cd8a48835d7e535dec779dc157"
Oct 31 19:23:43.120346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120312    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"]
Oct 31 19:23:43.120777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120751    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="extract"
Oct 31 19:23:43.120777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120767    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="extract"
Oct 31 19:23:43.120844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120780    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="util"
Oct 31 19:23:43.120844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120785    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="util"
Oct 31 19:23:43.120844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120792    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="pull"
Oct 31 19:23:43.120844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120797    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="pull"
Oct 31 19:23:43.120962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.120851    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f58e34e-789b-43c4-b4a3-e4fa843cf943" containerName="extract"
Oct 31 19:23:43.124112 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.124097    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.135400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.135378    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:23:43.135528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.135488    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 19:23:43.144307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.144284    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dprf8\""
Oct 31 19:23:43.145353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.145333    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"]
Oct 31 19:23:43.221827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.221793    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7c59x\" (UID: \"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.221827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.221840    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjhq\" (UniqueName: \"kubernetes.io/projected/30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340-kube-api-access-9hjhq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7c59x\" (UID: \"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.323202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.323158    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7c59x\" (UID: \"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.323359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.323218    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjhq\" (UniqueName: \"kubernetes.io/projected/30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340-kube-api-access-9hjhq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7c59x\" (UID: \"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.323582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.323532    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7c59x\" (UID: \"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.341405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.341373    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjhq\" (UniqueName: \"kubernetes.io/projected/30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340-kube-api-access-9hjhq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7c59x\" (UID: \"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.435340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.435249    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"
Oct 31 19:23:43.720226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:43.720200    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x"]
Oct 31 19:23:43.722905 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:23:43.722872    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30da9b3c_d6f9_4dad_ac3f_4eb2d3e0e340.slice/crio-9156c5d128366575f966fb08485544f123fa23bb210dcc9c2a32879e338a9665 WatchSource:0}: Error finding container 9156c5d128366575f966fb08485544f123fa23bb210dcc9c2a32879e338a9665: Status 404 returned error can't find the container with id 9156c5d128366575f966fb08485544f123fa23bb210dcc9c2a32879e338a9665
Oct 31 19:23:44.659942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:44.659907    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x" event={"ID":"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340","Type":"ContainerStarted","Data":"9156c5d128366575f966fb08485544f123fa23bb210dcc9c2a32879e338a9665"}
Oct 31 19:23:46.665360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:46.665319    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x" event={"ID":"30da9b3c-d6f9-4dad-ac3f-4eb2d3e0e340","Type":"ContainerStarted","Data":"37b668a20fb23ae422991a7ee074e945222cfc212b4f22c1838a0abd0b4a5b38"}
Oct 31 19:23:46.695434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:46.695377    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7c59x" podStartSLOduration=1.203991443 podStartE2EDuration="3.695358923s" totalImagesPullingTime="2.49136748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:46.694819426 +0000 UTC m=+220.156194045" watchObservedRunningTime="2026-10-31 19:23:46.695358923 +0000 UTC m=+220.156733540"
Oct 31 19:23:47.044277 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:23:47.044188    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:23:48.061920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.061887    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"]
Oct 31 19:23:48.065390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.065374    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.076693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.076657    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 19:23:48.076826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.076657    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5qp2d\""
Oct 31 19:23:48.076826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.076744    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 19:23:48.077713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.077679    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"]
Oct 31 19:23:48.170767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.170733    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75c5a1ce-03a1-44dd-a80a-72e6addba5f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2fbkm\" (UID: \"75c5a1ce-03a1-44dd-a80a-72e6addba5f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.170940 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.170793    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82qx\" (UniqueName: \"kubernetes.io/projected/75c5a1ce-03a1-44dd-a80a-72e6addba5f8-kube-api-access-v82qx\") pod \"cert-manager-webhook-5dbbbf75cc-2fbkm\" (UID: \"75c5a1ce-03a1-44dd-a80a-72e6addba5f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.271762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.271723    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75c5a1ce-03a1-44dd-a80a-72e6addba5f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2fbkm\" (UID: \"75c5a1ce-03a1-44dd-a80a-72e6addba5f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.271988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.271781    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82qx\" (UniqueName: \"kubernetes.io/projected/75c5a1ce-03a1-44dd-a80a-72e6addba5f8-kube-api-access-v82qx\") pod \"cert-manager-webhook-5dbbbf75cc-2fbkm\" (UID: \"75c5a1ce-03a1-44dd-a80a-72e6addba5f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.293750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.293708    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75c5a1ce-03a1-44dd-a80a-72e6addba5f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2fbkm\" (UID: \"75c5a1ce-03a1-44dd-a80a-72e6addba5f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.294683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.294661    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82qx\" (UniqueName: \"kubernetes.io/projected/75c5a1ce-03a1-44dd-a80a-72e6addba5f8-kube-api-access-v82qx\") pod \"cert-manager-webhook-5dbbbf75cc-2fbkm\" (UID: \"75c5a1ce-03a1-44dd-a80a-72e6addba5f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.385274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.385240    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:48.667890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:48.667865    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"]
Oct 31 19:23:48.670341 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:23:48.670287    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75c5a1ce_03a1_44dd_a80a_72e6addba5f8.slice/crio-c14854fc8850211024ed1d12d4366b5330255223e15dd5df77e3d2288534357e WatchSource:0}: Error finding container c14854fc8850211024ed1d12d4366b5330255223e15dd5df77e3d2288534357e: Status 404 returned error can't find the container with id c14854fc8850211024ed1d12d4366b5330255223e15dd5df77e3d2288534357e
Oct 31 19:23:49.679220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:49.679190    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm" event={"ID":"75c5a1ce-03a1-44dd-a80a-72e6addba5f8","Type":"ContainerStarted","Data":"c14854fc8850211024ed1d12d4366b5330255223e15dd5df77e3d2288534357e"}
Oct 31 19:23:52.284513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.284480    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"]
Oct 31 19:23:52.287860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.287843    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.293325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.293302    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fb7lf\""
Oct 31 19:23:52.299104 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.299081    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"]
Oct 31 19:23:52.415377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.415340    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f74r\" (UniqueName: \"kubernetes.io/projected/ba7bb823-557b-4e2b-a651-f0600af4ea50-kube-api-access-8f74r\") pod \"cert-manager-cainjector-5c57bbd4fc-z6nk5\" (UID: \"ba7bb823-557b-4e2b-a651-f0600af4ea50\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.415599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.415404    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba7bb823-557b-4e2b-a651-f0600af4ea50-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z6nk5\" (UID: \"ba7bb823-557b-4e2b-a651-f0600af4ea50\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.516471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.516428    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f74r\" (UniqueName: \"kubernetes.io/projected/ba7bb823-557b-4e2b-a651-f0600af4ea50-kube-api-access-8f74r\") pod \"cert-manager-cainjector-5c57bbd4fc-z6nk5\" (UID: \"ba7bb823-557b-4e2b-a651-f0600af4ea50\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.516668 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.516492    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba7bb823-557b-4e2b-a651-f0600af4ea50-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z6nk5\" (UID: \"ba7bb823-557b-4e2b-a651-f0600af4ea50\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.531194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.531161    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f74r\" (UniqueName: \"kubernetes.io/projected/ba7bb823-557b-4e2b-a651-f0600af4ea50-kube-api-access-8f74r\") pod \"cert-manager-cainjector-5c57bbd4fc-z6nk5\" (UID: \"ba7bb823-557b-4e2b-a651-f0600af4ea50\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.532063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.532043    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba7bb823-557b-4e2b-a651-f0600af4ea50-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z6nk5\" (UID: \"ba7bb823-557b-4e2b-a651-f0600af4ea50\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.600620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.600504    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"
Oct 31 19:23:52.693290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.693256    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm" event={"ID":"75c5a1ce-03a1-44dd-a80a-72e6addba5f8","Type":"ContainerStarted","Data":"582248bb9804d620350bdc1f08d441ab7eb8ae51f37605fa463199959d20ce44"}
Oct 31 19:23:52.693501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.693316    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:23:52.715923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.715875    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm" podStartSLOduration=1.719037621 podStartE2EDuration="4.715860151s" totalImagesPullingTime="2.99682253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:52.715325472 +0000 UTC m=+226.176700089" watchObservedRunningTime="2026-10-31 19:23:52.715860151 +0000 UTC m=+226.177234766"
Oct 31 19:23:52.880417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:52.880392    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5"]
Oct 31 19:23:52.882605 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:23:52.882576    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7bb823_557b_4e2b_a651_f0600af4ea50.slice/crio-191f555ba07b968217ec75c85f0921a0f1bfe36206c5295070abd40099c49710 WatchSource:0}: Error finding container 191f555ba07b968217ec75c85f0921a0f1bfe36206c5295070abd40099c49710: Status 404 returned error can't find the container with id 191f555ba07b968217ec75c85f0921a0f1bfe36206c5295070abd40099c49710
Oct 31 19:23:53.699307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:53.699266    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5" event={"ID":"ba7bb823-557b-4e2b-a651-f0600af4ea50","Type":"ContainerStarted","Data":"acffdf14388d70a955f54b12d6591da7198b72a363d892fcc407c83947412430"}
Oct 31 19:23:53.699732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:53.699317    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5" event={"ID":"ba7bb823-557b-4e2b-a651-f0600af4ea50","Type":"ContainerStarted","Data":"191f555ba07b968217ec75c85f0921a0f1bfe36206c5295070abd40099c49710"}
Oct 31 19:23:53.721194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:53.721142    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z6nk5" podStartSLOduration=1.721129104 podStartE2EDuration="1.721129104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:53.721022526 +0000 UTC m=+227.182397141" watchObservedRunningTime="2026-10-31 19:23:53.721129104 +0000 UTC m=+227.182503719"
Oct 31 19:23:56.680459 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.680424    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"]
Oct 31 19:23:56.683860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.683844    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.750204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.750168    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"]
Oct 31 19:23:56.751362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.751340    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 19:23:56.752127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.752111    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 19:23:56.752684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.752661    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 19:23:56.752805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.752686    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 19:23:56.752805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.752701    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 19:23:56.752805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.752723    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hzfp2\""
Oct 31 19:23:56.758204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.758177    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.758299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.758228    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.758346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.758294    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94078d47-127e-41bc-9bad-4151d823f923-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.758346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.758317    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szpf\" (UniqueName: \"kubernetes.io/projected/94078d47-127e-41bc-9bad-4151d823f923-kube-api-access-8szpf\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.758451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.758433    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.758509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.758495    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94078d47-127e-41bc-9bad-4151d823f923-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859047    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859263 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859247    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859297    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94078d47-127e-41bc-9bad-4151d823f923-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859326    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szpf\" (UniqueName: \"kubernetes.io/projected/94078d47-127e-41bc-9bad-4151d823f923-kube-api-access-8szpf\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859407    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859446    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94078d47-127e-41bc-9bad-4151d823f923-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.859792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.859721    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/94078d47-127e-41bc-9bad-4151d823f923-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.862017 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.861985    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94078d47-127e-41bc-9bad-4151d823f923-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.862240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.862221    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.863284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.863261    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.863778 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.863756    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/94078d47-127e-41bc-9bad-4151d823f923-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.887757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.887727    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szpf\" (UniqueName: \"kubernetes.io/projected/94078d47-127e-41bc-9bad-4151d823f923-kube-api-access-8szpf\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-dj7j4\" (UID: \"94078d47-127e-41bc-9bad-4151d823f923\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:56.999042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:56.998954    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:23:57.421037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:57.421002    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"]
Oct 31 19:23:57.421473 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:23:57.421439    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94078d47_127e_41bc_9bad_4151d823f923.slice/crio-666620254c2e2a72a356a5160b16c4baae6f0319e6c48c04a540fe6d6d7893dc WatchSource:0}: Error finding container 666620254c2e2a72a356a5160b16c4baae6f0319e6c48c04a540fe6d6d7893dc: Status 404 returned error can't find the container with id 666620254c2e2a72a356a5160b16c4baae6f0319e6c48c04a540fe6d6d7893dc
Oct 31 19:23:57.719660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:57.719578    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4" event={"ID":"94078d47-127e-41bc-9bad-4151d823f923","Type":"ContainerStarted","Data":"666620254c2e2a72a356a5160b16c4baae6f0319e6c48c04a540fe6d6d7893dc"}
Oct 31 19:23:58.702434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:23:58.702403    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm"
Oct 31 19:24:00.736406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:00.736364    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4" event={"ID":"94078d47-127e-41bc-9bad-4151d823f923","Type":"ContainerStarted","Data":"eaa0d7380bba4976cc30891ab14daeaaaff1ada1564408ed2d6a50b82805231c"}
Oct 31 19:24:01.742331 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:01.742293    5958 generic.go:356] "Generic (PLEG): container finished" podID="94078d47-127e-41bc-9bad-4151d823f923" containerID="eaa0d7380bba4976cc30891ab14daeaaaff1ada1564408ed2d6a50b82805231c" exitCode=0
Oct 31 19:24:01.742779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:01.742383    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4" event={"ID":"94078d47-127e-41bc-9bad-4151d823f923","Type":"ContainerDied","Data":"eaa0d7380bba4976cc30891ab14daeaaaff1ada1564408ed2d6a50b82805231c"}
Oct 31 19:24:02.749305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:02.749268    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4" event={"ID":"94078d47-127e-41bc-9bad-4151d823f923","Type":"ContainerStarted","Data":"5d20de8dcf242b4c47dfecafe82ea79c84c11ee6b615517ee187aed4f5488a91"}
Oct 31 19:24:02.749726 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:02.749357    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:24:02.802745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:02.802686    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4" podStartSLOduration=3.102032797 podStartE2EDuration="6.802670673s" totalImagesPullingTime="3.218679752s" totalInitContainerRuntime="481.958124ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:02.802137184 +0000 UTC m=+236.263511800" watchObservedRunningTime="2026-10-31 19:24:02.802670673 +0000 UTC m=+236.264045354"
Oct 31 19:24:07.908601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:07.908563    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"]
Oct 31 19:24:07.956285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:07.956242    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"]
Oct 31 19:24:07.956510 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:07.956386    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:07.969938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:07.969913    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 19:24:07.970065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:07.969973    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-h4nlg\""
Oct 31 19:24:07.975579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:07.975535    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:24:08.069687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.069653    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkl6m\" (UniqueName: \"kubernetes.io/projected/0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f-kube-api-access-fkl6m\") pod \"jobset-operator-674cd59dc5-ng96x\" (UID: \"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.069872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.069718    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f-tmp\") pod \"jobset-operator-674cd59dc5-ng96x\" (UID: \"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.170625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.170505    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkl6m\" (UniqueName: \"kubernetes.io/projected/0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f-kube-api-access-fkl6m\") pod \"jobset-operator-674cd59dc5-ng96x\" (UID: \"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.170806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.170673    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f-tmp\") pod \"jobset-operator-674cd59dc5-ng96x\" (UID: \"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.171040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.171024    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f-tmp\") pod \"jobset-operator-674cd59dc5-ng96x\" (UID: \"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.184525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.184492    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkl6m\" (UniqueName: \"kubernetes.io/projected/0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f-kube-api-access-fkl6m\") pod \"jobset-operator-674cd59dc5-ng96x\" (UID: \"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.284097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.284053    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"
Oct 31 19:24:08.588326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.588300    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x"]
Oct 31 19:24:08.591331 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:24:08.591306    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d7e46e2_5d40_4ecf_8a40_5b81d12f9a4f.slice/crio-ecbf9222a000c55ff4c802ec99dce5ec2e86472e1fe16caa0967e98a540e1164 WatchSource:0}: Error finding container ecbf9222a000c55ff4c802ec99dce5ec2e86472e1fe16caa0967e98a540e1164: Status 404 returned error can't find the container with id ecbf9222a000c55ff4c802ec99dce5ec2e86472e1fe16caa0967e98a540e1164
Oct 31 19:24:08.775865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:08.775831    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x" event={"ID":"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f","Type":"ContainerStarted","Data":"ecbf9222a000c55ff4c802ec99dce5ec2e86472e1fe16caa0967e98a540e1164"}
Oct 31 19:24:11.791245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:11.791202    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x" event={"ID":"0d7e46e2-5d40-4ecf-8a40-5b81d12f9a4f","Type":"ContainerStarted","Data":"90c8bc88af56c925e4afa7acd87c71811d5ed791e787f919eec2db5692ccd982"}
Oct 31 19:24:11.830286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:11.830220    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ng96x" podStartSLOduration=1.73361509 podStartE2EDuration="4.830205323s" totalImagesPullingTime="3.096590233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:11.829796179 +0000 UTC m=+245.291170794" watchObservedRunningTime="2026-10-31 19:24:11.830205323 +0000 UTC m=+245.291579937"
Oct 31 19:24:13.756078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:13.756045    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4"
Oct 31 19:24:17.043358 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:24:17.043314    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:24:24.687365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.687329    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"]
Oct 31 19:24:24.691782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.691755    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.702047 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.702021    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 19:24:24.702047 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.702042    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-skfzl\""
Oct 31 19:24:24.702213 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.702042    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 19:24:24.705502 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.705484    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 19:24:24.718280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.718248    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssjq\" (UniqueName: \"kubernetes.io/projected/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-kube-api-access-nssjq\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.718454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.718314    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-tls-certs\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.718454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.718345    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-tmp\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.720206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.720184    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"]
Oct 31 19:24:24.807223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.807195    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 19:24:24.819866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.819839    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssjq\" (UniqueName: \"kubernetes.io/projected/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-kube-api-access-nssjq\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.820038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.819899    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-tls-certs\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.820038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.819932    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-tmp\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.822436 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.822402    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-tmp\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.822791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.822766    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-tls-certs\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:24.844129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:24.844100    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssjq\" (UniqueName: \"kubernetes.io/projected/9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43-kube-api-access-nssjq\") pod \"kube-auth-proxy-849d94dfbb-ff72l\" (UID: \"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:25.005252 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:25.005161    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"
Oct 31 19:24:25.292854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:25.292830    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l"]
Oct 31 19:24:25.295527 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:24:25.295497    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ba7c4_8fd2_441f_9045_7e2bc9fdda43.slice/crio-92a830bb9ade1c70a47bcac3e4b2ca14b4debd7139548edaabc2f0825d287332 WatchSource:0}: Error finding container 92a830bb9ade1c70a47bcac3e4b2ca14b4debd7139548edaabc2f0825d287332: Status 404 returned error can't find the container with id 92a830bb9ade1c70a47bcac3e4b2ca14b4debd7139548edaabc2f0825d287332
Oct 31 19:24:25.852196 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:25.852161    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l" event={"ID":"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43","Type":"ContainerStarted","Data":"92a830bb9ade1c70a47bcac3e4b2ca14b4debd7139548edaabc2f0825d287332"}
Oct 31 19:24:26.363128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.361273    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:26.366385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.365016    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.379610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.379577    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:26.391954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.391851    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 19:24:26.391954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.391872    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 19:24:26.391954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.391886    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mwdrn\""
Oct 31 19:24:26.391954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.391929    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 19:24:26.434609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.434569    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file\") pod \"limitador-limitador-74dfcfb894-m74sp\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.434754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.434632    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qb5j\" (UniqueName: \"kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j\") pod \"limitador-limitador-74dfcfb894-m74sp\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.465762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.465731    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:26.535391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.535359    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file\") pod \"limitador-limitador-74dfcfb894-m74sp\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.535583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.535415    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qb5j\" (UniqueName: \"kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j\") pod \"limitador-limitador-74dfcfb894-m74sp\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.536078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.536059    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file\") pod \"limitador-limitador-74dfcfb894-m74sp\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.551918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.551889    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qb5j\" (UniqueName: \"kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j\") pod \"limitador-limitador-74dfcfb894-m74sp\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.685675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.685635    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:26.978917 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:26.978887    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:26.981644 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:24:26.981615    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a009e0_2578_4086_83ff_85199da35720.slice/crio-e1c2fff0355782ef8aaf321a85a30e69a107fff87f1666b6c51cada09a9d74b4 WatchSource:0}: Error finding container e1c2fff0355782ef8aaf321a85a30e69a107fff87f1666b6c51cada09a9d74b4: Status 404 returned error can't find the container with id e1c2fff0355782ef8aaf321a85a30e69a107fff87f1666b6c51cada09a9d74b4
Oct 31 19:24:27.252625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.252501    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:24:27.263965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.263933    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.279841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.279812    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:24:27.321813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.321769    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:24:27.342091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.342060    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d10c6766-069b-4096-9b7b-67ce81c55dd9-config-file\") pod \"limitador-limitador-7c9b87d574-9vnsw\" (UID: \"d10c6766-069b-4096-9b7b-67ce81c55dd9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.342267 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.342179    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24x5\" (UniqueName: \"kubernetes.io/projected/d10c6766-069b-4096-9b7b-67ce81c55dd9-kube-api-access-l24x5\") pod \"limitador-limitador-7c9b87d574-9vnsw\" (UID: \"d10c6766-069b-4096-9b7b-67ce81c55dd9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.444769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.443572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d10c6766-069b-4096-9b7b-67ce81c55dd9-config-file\") pod \"limitador-limitador-7c9b87d574-9vnsw\" (UID: \"d10c6766-069b-4096-9b7b-67ce81c55dd9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.444769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.443643    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24x5\" (UniqueName: \"kubernetes.io/projected/d10c6766-069b-4096-9b7b-67ce81c55dd9-kube-api-access-l24x5\") pod \"limitador-limitador-7c9b87d574-9vnsw\" (UID: \"d10c6766-069b-4096-9b7b-67ce81c55dd9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.444769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.444700    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d10c6766-069b-4096-9b7b-67ce81c55dd9-config-file\") pod \"limitador-limitador-7c9b87d574-9vnsw\" (UID: \"d10c6766-069b-4096-9b7b-67ce81c55dd9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.479561 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.479516    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24x5\" (UniqueName: \"kubernetes.io/projected/d10c6766-069b-4096-9b7b-67ce81c55dd9-kube-api-access-l24x5\") pod \"limitador-limitador-7c9b87d574-9vnsw\" (UID: \"d10c6766-069b-4096-9b7b-67ce81c55dd9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.594760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.594674    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:27.863994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.863958    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l" event={"ID":"9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43","Type":"ContainerStarted","Data":"d3e19aa4c407bab5625dbf813b6a54c7bb1fe51be4213d07fb844463342921fb"}
Oct 31 19:24:27.865311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.865284    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp" event={"ID":"c4a009e0-2578-4086-83ff-85199da35720","Type":"ContainerStarted","Data":"e1c2fff0355782ef8aaf321a85a30e69a107fff87f1666b6c51cada09a9d74b4"}
Oct 31 19:24:27.885180 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.885156    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:24:27.887270 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:24:27.887241    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10c6766_069b_4096_9b7b_67ce81c55dd9.slice/crio-52b23a4c072294276a416eb2d406bf7616059d5f2f9ec109cd0fbe63ed6832a2 WatchSource:0}: Error finding container 52b23a4c072294276a416eb2d406bf7616059d5f2f9ec109cd0fbe63ed6832a2: Status 404 returned error can't find the container with id 52b23a4c072294276a416eb2d406bf7616059d5f2f9ec109cd0fbe63ed6832a2
Oct 31 19:24:27.891170 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:27.891127    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-ff72l" podStartSLOduration=1.396320375 podStartE2EDuration="3.891115287s" totalImagesPullingTime="2.494794912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:27.890856202 +0000 UTC m=+261.352230818" watchObservedRunningTime="2026-10-31 19:24:27.891115287 +0000 UTC m=+261.352489899"
Oct 31 19:24:28.873717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:28.873647    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw" event={"ID":"d10c6766-069b-4096-9b7b-67ce81c55dd9","Type":"ContainerStarted","Data":"52b23a4c072294276a416eb2d406bf7616059d5f2f9ec109cd0fbe63ed6832a2"}
Oct 31 19:24:30.885574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:30.885518    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw" event={"ID":"d10c6766-069b-4096-9b7b-67ce81c55dd9","Type":"ContainerStarted","Data":"cb53f806dbdb5a9a3205725d6ae585c0fc1bc3536fcd46f3de2e02baf8cde558"}
Oct 31 19:24:30.886055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:30.885611    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:30.887288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:30.887263    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp" event={"ID":"c4a009e0-2578-4086-83ff-85199da35720","Type":"ContainerStarted","Data":"a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c"}
Oct 31 19:24:30.887408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:30.887320    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:30.908378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:30.908264    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw" podStartSLOduration=1.258156003 podStartE2EDuration="3.908250245s" totalImagesPullingTime="2.650094242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:30.9059628 +0000 UTC m=+264.367337416" watchObservedRunningTime="2026-10-31 19:24:30.908250245 +0000 UTC m=+264.369624859"
Oct 31 19:24:30.931487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:30.931436    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp" podStartSLOduration=1.76731671 podStartE2EDuration="4.931421961s" totalImagesPullingTime="3.164105251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:30.930872494 +0000 UTC m=+264.392247112" watchObservedRunningTime="2026-10-31 19:24:30.931421961 +0000 UTC m=+264.392796572"
Oct 31 19:24:41.894143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:41.894109    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:41.894698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:41.894317    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"
Oct 31 19:24:41.998842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:41.998813    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:41.999034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:41.998988    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp" podUID="c4a009e0-2578-4086-83ff-85199da35720" containerName="limitador" containerID="cri-o://a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c" gracePeriod=30
Oct 31 19:24:42.941433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.941402    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:42.944094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.944068    5958 generic.go:356] "Generic (PLEG): container finished" podID="c4a009e0-2578-4086-83ff-85199da35720" containerID="a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c" exitCode=0
Oct 31 19:24:42.944208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.944109    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp"
Oct 31 19:24:42.944208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.944104    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp" event={"ID":"c4a009e0-2578-4086-83ff-85199da35720","Type":"ContainerDied","Data":"a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c"}
Oct 31 19:24:42.944282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.944222    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-m74sp" event={"ID":"c4a009e0-2578-4086-83ff-85199da35720","Type":"ContainerDied","Data":"e1c2fff0355782ef8aaf321a85a30e69a107fff87f1666b6c51cada09a9d74b4"}
Oct 31 19:24:42.944282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.944248    5958 scope.go:122] "RemoveContainer" containerID="a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c"
Oct 31 19:24:42.952259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.952243    5958 scope.go:122] "RemoveContainer" containerID="a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c"
Oct 31 19:24:42.952522 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:24:42.952504    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c\": container with ID starting with a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c not found: ID does not exist" containerID="a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c"
Oct 31 19:24:42.952588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.952530    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c"} err="failed to get container status \"a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c\": rpc error: code = NotFound desc = could not find container \"a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c\": container with ID starting with a2e4b2a967280c04d4caf87b64c7a14b7c760d1f5c4e7ca62901a431db0def6c not found: ID does not exist"
Oct 31 19:24:42.999066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.999034    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file\" (UniqueName: \"kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file\") pod \"c4a009e0-2578-4086-83ff-85199da35720\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") "
Oct 31 19:24:42.999217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.999126    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j\" (UniqueName: \"kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j\") pod \"c4a009e0-2578-4086-83ff-85199da35720\" (UID: \"c4a009e0-2578-4086-83ff-85199da35720\") "
Oct 31 19:24:42.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:42.999351    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file" pod "c4a009e0-2578-4086-83ff-85199da35720" (UID: "c4a009e0-2578-4086-83ff-85199da35720"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:43.001297 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:43.001276    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j" pod "c4a009e0-2578-4086-83ff-85199da35720" (UID: "c4a009e0-2578-4086-83ff-85199da35720"). InnerVolumeSpecName "kube-api-access-2qb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:43.100233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:43.100149    5958 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4a009e0-2578-4086-83ff-85199da35720-config-file\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:24:43.100233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:43.100179    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qb5j\" (UniqueName: \"kubernetes.io/projected/c4a009e0-2578-4086-83ff-85199da35720-kube-api-access-2qb5j\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:24:43.268201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:43.268167    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:43.272308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:43.272273    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-m74sp"]
Oct 31 19:24:44.661410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:44.661376    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a009e0-2578-4086-83ff-85199da35720" path="/var/lib/kubelet/pods/c4a009e0-2578-4086-83ff-85199da35720/volumes"
Oct 31 19:24:47.046057 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:24:47.044406    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:24:47.887506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.887472    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"]
Oct 31 19:24:47.887925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.887910    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c4a009e0-2578-4086-83ff-85199da35720" containerName="limitador"
Oct 31 19:24:47.887971 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.887927    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a009e0-2578-4086-83ff-85199da35720" containerName="limitador"
Oct 31 19:24:47.888005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.887988    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a009e0-2578-4086-83ff-85199da35720" containerName="limitador"
Oct 31 19:24:47.890942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.890924    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:47.901654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.901626    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 19:24:47.901654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.901642    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:24:47.901841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.901803    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sz7bm\""
Oct 31 19:24:47.904128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.904105    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:24:47.905835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:47.905806    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"]
Oct 31 19:24:48.047258 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.047223    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cf4ca2ac-f190-45de-a720-bc15b2fe094a-data\") pod \"postgres-7cdbf6dc44-v4wx9\" (UID: \"cf4ca2ac-f190-45de-a720-bc15b2fe094a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.047258 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.047267    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzmp\" (UniqueName: \"kubernetes.io/projected/cf4ca2ac-f190-45de-a720-bc15b2fe094a-kube-api-access-lzzmp\") pod \"postgres-7cdbf6dc44-v4wx9\" (UID: \"cf4ca2ac-f190-45de-a720-bc15b2fe094a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.148785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.148680    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cf4ca2ac-f190-45de-a720-bc15b2fe094a-data\") pod \"postgres-7cdbf6dc44-v4wx9\" (UID: \"cf4ca2ac-f190-45de-a720-bc15b2fe094a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.148785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.148741    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzmp\" (UniqueName: \"kubernetes.io/projected/cf4ca2ac-f190-45de-a720-bc15b2fe094a-kube-api-access-lzzmp\") pod \"postgres-7cdbf6dc44-v4wx9\" (UID: \"cf4ca2ac-f190-45de-a720-bc15b2fe094a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.149107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.149083    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cf4ca2ac-f190-45de-a720-bc15b2fe094a-data\") pod \"postgres-7cdbf6dc44-v4wx9\" (UID: \"cf4ca2ac-f190-45de-a720-bc15b2fe094a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.162404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.162372    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzmp\" (UniqueName: \"kubernetes.io/projected/cf4ca2ac-f190-45de-a720-bc15b2fe094a-kube-api-access-lzzmp\") pod \"postgres-7cdbf6dc44-v4wx9\" (UID: \"cf4ca2ac-f190-45de-a720-bc15b2fe094a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.208345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.208310    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:48.495784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.495760    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"]
Oct 31 19:24:48.497877 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:24:48.497831    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4ca2ac_f190_45de_a720_bc15b2fe094a.slice/crio-c1324865d438395cb892f08dda353786a987003f096f21cfaefd63ea3511b85d WatchSource:0}: Error finding container c1324865d438395cb892f08dda353786a987003f096f21cfaefd63ea3511b85d: Status 404 returned error can't find the container with id c1324865d438395cb892f08dda353786a987003f096f21cfaefd63ea3511b85d
Oct 31 19:24:48.972080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:48.972040    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9" event={"ID":"cf4ca2ac-f190-45de-a720-bc15b2fe094a","Type":"ContainerStarted","Data":"c1324865d438395cb892f08dda353786a987003f096f21cfaefd63ea3511b85d"}
Oct 31 19:24:55.006650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:55.006613    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9" event={"ID":"cf4ca2ac-f190-45de-a720-bc15b2fe094a","Type":"ContainerStarted","Data":"c4a3480d97f934ffc51aa38dd22af2a12add3471403665d7609e2afcc07b8c5a"}
Oct 31 19:24:55.007094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:55.006763    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:24:55.027275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:24:55.027230    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9" podStartSLOduration=2.156469798 podStartE2EDuration="8.027218308s" totalImagesPullingTime="5.87074851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:55.026989708 +0000 UTC m=+288.488364326" watchObservedRunningTime="2026-10-31 19:24:55.027218308 +0000 UTC m=+288.488592923"
Oct 31 19:25:01.041863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:01.041832    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-v4wx9"
Oct 31 19:25:08.152248 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.152215    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwwwn"]
Oct 31 19:25:08.162507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.162483    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.169392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.169360    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v94n9\""
Oct 31 19:25:08.169392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.169372    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 19:25:08.169392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.169363    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 19:25:08.169392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.169372    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 19:25:08.171048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.171029    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 19:25:08.171691 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.171656    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 19:25:08.173458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.173436    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwwwn"]
Oct 31 19:25:08.235045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.235003    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5x6\" (UniqueName: \"kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.235234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.235129    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.235234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.235207    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.336309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.336264    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.336488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.336331    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.336488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.336372    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5x6\" (UniqueName: \"kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.339015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.338987    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.339015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.339005    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.350369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.350343    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5x6\" (UniqueName: \"kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6\") pod \"maas-controller-579cf7c6d-gwwwn\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") " pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.476560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.476475    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:08.494208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.494180    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8689db56c-vdnn8"]
Oct 31 19:25:08.506100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.506065    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.508040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.508011    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689db56c-vdnn8"]
Oct 31 19:25:08.538723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.538693    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.538856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.538735    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j766\" (UniqueName: \"kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.538916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.538841    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.640199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.640151    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.640381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.640213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j766\" (UniqueName: \"kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.640381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.640247    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.642777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.642751    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.642879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.642755    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.653835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.653812    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j766\" (UniqueName: \"kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766\") pod \"maas-controller-8689db56c-vdnn8\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") " pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:08.759341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.759278    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwwwn"]
Oct 31 19:25:08.761502 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:25:08.761469    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b870d2_0f39_4917_a49d_43392f3e7f90.slice/crio-37f4cd5bb17b07bd48067a4cf8ec36b4da3cdc4ac07b7aa62f22c8647f94a5a2 WatchSource:0}: Error finding container 37f4cd5bb17b07bd48067a4cf8ec36b4da3cdc4ac07b7aa62f22c8647f94a5a2: Status 404 returned error can't find the container with id 37f4cd5bb17b07bd48067a4cf8ec36b4da3cdc4ac07b7aa62f22c8647f94a5a2
Oct 31 19:25:08.824930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:08.824892    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:09.073389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:09.073301    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwwwn" event={"ID":"a1b870d2-0f39-4917-a49d-43392f3e7f90","Type":"ContainerStarted","Data":"37f4cd5bb17b07bd48067a4cf8ec36b4da3cdc4ac07b7aa62f22c8647f94a5a2"}
Oct 31 19:25:09.099976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:09.099951    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689db56c-vdnn8"]
Oct 31 19:25:09.102130 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:25:09.102098    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4818c270_d27d_4689_9ea9_49d3b0e56579.slice/crio-a468e1667bedcf4047cdc35f51455286ed34fdbe392b1c30e937c18dc3d3b652 WatchSource:0}: Error finding container a468e1667bedcf4047cdc35f51455286ed34fdbe392b1c30e937c18dc3d3b652: Status 404 returned error can't find the container with id a468e1667bedcf4047cdc35f51455286ed34fdbe392b1c30e937c18dc3d3b652
Oct 31 19:25:10.082111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:10.082080    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-vdnn8" event={"ID":"4818c270-d27d-4689-9ea9-49d3b0e56579","Type":"ContainerStarted","Data":"a468e1667bedcf4047cdc35f51455286ed34fdbe392b1c30e937c18dc3d3b652"}
Oct 31 19:25:12.093975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:12.093930    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-vdnn8" event={"ID":"4818c270-d27d-4689-9ea9-49d3b0e56579","Type":"ContainerStarted","Data":"24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"}
Oct 31 19:25:12.094437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:12.094100    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:12.095727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:12.095691    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwwwn" event={"ID":"a1b870d2-0f39-4917-a49d-43392f3e7f90","Type":"ContainerStarted","Data":"7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96"}
Oct 31 19:25:12.095853 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:12.095761    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:12.123663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:12.123606    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8689db56c-vdnn8" podStartSLOduration=1.31684962 podStartE2EDuration="4.123590404s" totalImagesPullingTime="2.806740784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:12.122003137 +0000 UTC m=+305.583377753" watchObservedRunningTime="2026-10-31 19:25:12.123590404 +0000 UTC m=+305.584965050"
Oct 31 19:25:12.145749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:12.145691    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-gwwwn" podStartSLOduration=1.361293141 podStartE2EDuration="4.145675562s" totalImagesPullingTime="2.784382421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:12.144622774 +0000 UTC m=+305.605997388" watchObservedRunningTime="2026-10-31 19:25:12.145675562 +0000 UTC m=+305.607050177"
Oct 31 19:25:13.810748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.810709    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"]
Oct 31 19:25:13.814304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.814283    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.836938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.836905    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 19:25:13.837266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.837249    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-lsrm5\""
Oct 31 19:25:13.837356 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.837337    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 19:25:13.837405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.837352    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 19:25:13.872375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.872339    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"]
Oct 31 19:25:13.893888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.893852    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f56ac52-327e-4e63-a190-346728a6e78c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.894060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.893895    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bfsz\" (UniqueName: \"kubernetes.io/projected/7f56ac52-327e-4e63-a190-346728a6e78c-kube-api-access-9bfsz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.894060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.894042    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f56ac52-327e-4e63-a190-346728a6e78c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.894159 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.894099    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7f56ac52-327e-4e63-a190-346728a6e78c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.995286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.995243    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfsz\" (UniqueName: \"kubernetes.io/projected/7f56ac52-327e-4e63-a190-346728a6e78c-kube-api-access-9bfsz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.995457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.995386    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f56ac52-327e-4e63-a190-346728a6e78c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.995499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.995451    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7f56ac52-327e-4e63-a190-346728a6e78c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.995620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.995505    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f56ac52-327e-4e63-a190-346728a6e78c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.995748 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:25:13.995718    5958 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 19:25:13.995876 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:25:13.995805    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f56ac52-327e-4e63-a190-346728a6e78c-tls-cert podName:7f56ac52-327e-4e63-a190-346728a6e78c nodeName:}" failed. No retries permitted until 2026-10-31 19:25:14.495784071 +0000 UTC m=+307.957158679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/7f56ac52-327e-4e63-a190-346728a6e78c-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-c79hl" (UID: "7f56ac52-327e-4e63-a190-346728a6e78c") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 31 19:25:13.996223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.996195    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f56ac52-327e-4e63-a190-346728a6e78c-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:13.996400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:13.996379    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7f56ac52-327e-4e63-a190-346728a6e78c-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:14.012001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:14.011965    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfsz\" (UniqueName: \"kubernetes.io/projected/7f56ac52-327e-4e63-a190-346728a6e78c-kube-api-access-9bfsz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:14.500246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:14.500202    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f56ac52-327e-4e63-a190-346728a6e78c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:14.502896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:14.502866    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7f56ac52-327e-4e63-a190-346728a6e78c-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-c79hl\" (UID: \"7f56ac52-327e-4e63-a190-346728a6e78c\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:14.728324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:14.728288    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:15.001056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:15.001031    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"]
Oct 31 19:25:15.003165 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:25:15.003135    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f56ac52_327e_4e63_a190_346728a6e78c.slice/crio-bffc5e3e77530fe964eb4bb64c3c5aefe5785707833d26d2abcbe0d40fbcbc73 WatchSource:0}: Error finding container bffc5e3e77530fe964eb4bb64c3c5aefe5785707833d26d2abcbe0d40fbcbc73: Status 404 returned error can't find the container with id bffc5e3e77530fe964eb4bb64c3c5aefe5785707833d26d2abcbe0d40fbcbc73
Oct 31 19:25:15.111568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:15.111460    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl" event={"ID":"7f56ac52-327e-4e63-a190-346728a6e78c","Type":"ContainerStarted","Data":"bffc5e3e77530fe964eb4bb64c3c5aefe5785707833d26d2abcbe0d40fbcbc73"}
Oct 31 19:25:17.050565 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:25:17.048640    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:25:20.824258 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.824226    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hbjrc"]
Oct 31 19:25:20.849598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.849563    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hbjrc"]
Oct 31 19:25:20.849782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.849666    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:20.856701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.856673    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ckkqg\""
Oct 31 19:25:20.861014 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.860969    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 19:25:20.907694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.907654    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-v2zwj"]
Oct 31 19:25:20.911809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.911782    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:20.927811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.927775    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-v2zwj"]
Oct 31 19:25:20.961357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.961328    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 19:25:20.961606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.961385    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 19:25:20.961606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.961405    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-c2q2v\""
Oct 31 19:25:20.966067 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.966034    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-cert\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:20.966223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.966093    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmh4\" (UniqueName: \"kubernetes.io/projected/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-kube-api-access-sdmh4\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:20.966288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.966232    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c81050c1-9e46-4570-b1a5-c9faa5de4014-cert\") pod \"kserve-controller-manager-776cc66d47-hbjrc\" (UID: \"c81050c1-9e46-4570-b1a5-c9faa5de4014\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:20.966351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.966300    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2pld\" (UniqueName: \"kubernetes.io/projected/c81050c1-9e46-4570-b1a5-c9faa5de4014-kube-api-access-c2pld\") pod \"kserve-controller-manager-776cc66d47-hbjrc\" (UID: \"c81050c1-9e46-4570-b1a5-c9faa5de4014\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:20.966351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:20.966343    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-metrics-certs\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.067701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.067656    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-cert\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.067892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.067716    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmh4\" (UniqueName: \"kubernetes.io/projected/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-kube-api-access-sdmh4\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.067892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.067755    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c81050c1-9e46-4570-b1a5-c9faa5de4014-cert\") pod \"kserve-controller-manager-776cc66d47-hbjrc\" (UID: \"c81050c1-9e46-4570-b1a5-c9faa5de4014\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:21.067892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.067816    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2pld\" (UniqueName: \"kubernetes.io/projected/c81050c1-9e46-4570-b1a5-c9faa5de4014-kube-api-access-c2pld\") pod \"kserve-controller-manager-776cc66d47-hbjrc\" (UID: \"c81050c1-9e46-4570-b1a5-c9faa5de4014\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:21.067892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.067841    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-metrics-certs\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.070932 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.070898    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-metrics-certs\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.070932 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.070929    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c81050c1-9e46-4570-b1a5-c9faa5de4014-cert\") pod \"kserve-controller-manager-776cc66d47-hbjrc\" (UID: \"c81050c1-9e46-4570-b1a5-c9faa5de4014\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:21.071145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.070948    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-cert\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.082490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.082412    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmh4\" (UniqueName: \"kubernetes.io/projected/a0bd8286-fc8a-46e4-9c81-ca98bd386d0d-kube-api-access-sdmh4\") pod \"odh-model-controller-769875db56-v2zwj\" (UID: \"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d\") " pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.083033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.083012    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2pld\" (UniqueName: \"kubernetes.io/projected/c81050c1-9e46-4570-b1a5-c9faa5de4014-kube-api-access-c2pld\") pod \"kserve-controller-manager-776cc66d47-hbjrc\" (UID: \"c81050c1-9e46-4570-b1a5-c9faa5de4014\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:21.149263 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.149226    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"]
Oct 31 19:25:21.157150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.157120    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.166582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.166537    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 19:25:21.166756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.166612    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 19:25:21.166948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.166829    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 19:25:21.166948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.166701    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 19:25:21.166948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.166913    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7z5vd\""
Oct 31 19:25:21.168160 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.168135    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:21.168879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.168853    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"]
Oct 31 19:25:21.231824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.231470    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269199    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269296    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269347    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269381    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269404    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269449    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.271608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.269477    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5cl\" (UniqueName: \"kubernetes.io/projected/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-kube-api-access-pf5cl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370332    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370434    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370489    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370525    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370560    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.370701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.370625    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5cl\" (UniqueName: \"kubernetes.io/projected/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-kube-api-access-pf5cl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.371840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.371786    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.375399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.375338    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.375529 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.375494    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.377787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.377334    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.377787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.377422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.395493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.395456    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5cl\" (UniqueName: \"kubernetes.io/projected/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-kube-api-access-pf5cl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.422020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.421960    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzhhr\" (UID: \"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.473889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.473613    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:21.477306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.477276    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hbjrc"]
Oct 31 19:25:21.479394 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:25:21.479363    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81050c1_9e46_4570_b1a5_c9faa5de4014.slice/crio-790d1ad6597b08f074af38782baa9faf333b353ca7982b8f760d74e7a289fd6e WatchSource:0}: Error finding container 790d1ad6597b08f074af38782baa9faf333b353ca7982b8f760d74e7a289fd6e: Status 404 returned error can't find the container with id 790d1ad6597b08f074af38782baa9faf333b353ca7982b8f760d74e7a289fd6e
Oct 31 19:25:21.556419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.552609    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-v2zwj"]
Oct 31 19:25:21.558999 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:25:21.558960    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bd8286_fc8a_46e4_9c81_ca98bd386d0d.slice/crio-f406b7086ed0b2f851d906e9d36be7d3e5317f962413b005e901d1dc8ea962b7 WatchSource:0}: Error finding container f406b7086ed0b2f851d906e9d36be7d3e5317f962413b005e901d1dc8ea962b7: Status 404 returned error can't find the container with id f406b7086ed0b2f851d906e9d36be7d3e5317f962413b005e901d1dc8ea962b7
Oct 31 19:25:21.772555 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:21.772454    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"]
Oct 31 19:25:21.776623 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:25:21.776586    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0fa91f_68e3_4ba1_8cc7_e9a4d1704757.slice/crio-171d2d9569a44f9304a0f742c7f2f1df96644ae2ee613034e74f03024e8fbbcc WatchSource:0}: Error finding container 171d2d9569a44f9304a0f742c7f2f1df96644ae2ee613034e74f03024e8fbbcc: Status 404 returned error can't find the container with id 171d2d9569a44f9304a0f742c7f2f1df96644ae2ee613034e74f03024e8fbbcc
Oct 31 19:25:22.156070 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:22.156031    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr" event={"ID":"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757","Type":"ContainerStarted","Data":"171d2d9569a44f9304a0f742c7f2f1df96644ae2ee613034e74f03024e8fbbcc"}
Oct 31 19:25:22.158038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:22.158004    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-v2zwj" event={"ID":"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d","Type":"ContainerStarted","Data":"f406b7086ed0b2f851d906e9d36be7d3e5317f962413b005e901d1dc8ea962b7"}
Oct 31 19:25:22.159717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:22.159658    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc" event={"ID":"c81050c1-9e46-4570-b1a5-c9faa5de4014","Type":"ContainerStarted","Data":"790d1ad6597b08f074af38782baa9faf333b353ca7982b8f760d74e7a289fd6e"}
Oct 31 19:25:23.105197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.103714    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:25:23.105197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.103814    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:23.186032 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.185985    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwwwn"]
Oct 31 19:25:23.187912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.187870    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-gwwwn" podUID="a1b870d2-0f39-4917-a49d-43392f3e7f90" containerName="manager" containerID="cri-o://7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96" gracePeriod=30
Oct 31 19:25:23.564449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.564178    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:23.594136 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.594071    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert\") pod \"a1b870d2-0f39-4917-a49d-43392f3e7f90\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") "
Oct 31 19:25:23.594325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.594152    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6\" (UniqueName: \"kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6\") pod \"a1b870d2-0f39-4917-a49d-43392f3e7f90\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") "
Oct 31 19:25:23.594325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.594277    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs\") pod \"a1b870d2-0f39-4917-a49d-43392f3e7f90\" (UID: \"a1b870d2-0f39-4917-a49d-43392f3e7f90\") "
Oct 31 19:25:23.604028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.603970    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6" pod "a1b870d2-0f39-4917-a49d-43392f3e7f90" (UID: "a1b870d2-0f39-4917-a49d-43392f3e7f90"). InnerVolumeSpecName "kube-api-access-ff5x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:23.604393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.604347    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert" pod "a1b870d2-0f39-4917-a49d-43392f3e7f90" (UID: "a1b870d2-0f39-4917-a49d-43392f3e7f90"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:23.613849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.613806    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs" pod "a1b870d2-0f39-4917-a49d-43392f3e7f90" (UID: "a1b870d2-0f39-4917-a49d-43392f3e7f90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:23.696305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.696261    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:23.696305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.696307    5958 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b870d2-0f39-4917-a49d-43392f3e7f90-webhook-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:23.696509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:23.696323    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5x6\" (UniqueName: \"kubernetes.io/projected/a1b870d2-0f39-4917-a49d-43392f3e7f90-kube-api-access-ff5x6\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:24.176758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.176718    5958 generic.go:356] "Generic (PLEG): container finished" podID="a1b870d2-0f39-4917-a49d-43392f3e7f90" containerID="7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96" exitCode=0
Oct 31 19:25:24.176955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.176785    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwwwn" event={"ID":"a1b870d2-0f39-4917-a49d-43392f3e7f90","Type":"ContainerDied","Data":"7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96"}
Oct 31 19:25:24.176955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.176869    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-gwwwn" event={"ID":"a1b870d2-0f39-4917-a49d-43392f3e7f90","Type":"ContainerDied","Data":"37f4cd5bb17b07bd48067a4cf8ec36b4da3cdc4ac07b7aa62f22c8647f94a5a2"}
Oct 31 19:25:24.176955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.176869    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-gwwwn"
Oct 31 19:25:24.176955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.176898    5958 scope.go:122] "RemoveContainer" containerID="7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96"
Oct 31 19:25:24.198309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.198091    5958 scope.go:122] "RemoveContainer" containerID="7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96"
Oct 31 19:25:24.198774 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:25:24.198489    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96\": container with ID starting with 7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96 not found: ID does not exist" containerID="7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96"
Oct 31 19:25:24.198774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.198530    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96"} err="failed to get container status \"7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96\": rpc error: code = NotFound desc = could not find container \"7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96\": container with ID starting with 7ac1b2cc529816d14f60171ea3f393ba641cc4e9132eccec6f70ca917a51fe96 not found: ID does not exist"
Oct 31 19:25:24.207929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.207878    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwwwn"]
Oct 31 19:25:24.211362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.211336    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-gwwwn"]
Oct 31 19:25:24.664807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:24.664766    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b870d2-0f39-4917-a49d-43392f3e7f90" path="/var/lib/kubelet/pods/a1b870d2-0f39-4917-a49d-43392f3e7f90/volumes"
Oct 31 19:25:25.189372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:25.188931    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc" event={"ID":"c81050c1-9e46-4570-b1a5-c9faa5de4014","Type":"ContainerStarted","Data":"78a347896c12b85940aed1d5f9c55423836b47d49f0c06d61bf17af0c64b4d09"}
Oct 31 19:25:25.189699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:25.189677    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:25:25.211406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:25.211336    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc" podStartSLOduration=2.010964913 podStartE2EDuration="5.211315427s" totalImagesPullingTime="3.200350514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:25.210811996 +0000 UTC m=+318.672186612" watchObservedRunningTime="2026-10-31 19:25:25.211315427 +0000 UTC m=+318.672690039"
Oct 31 19:25:26.419843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:26.419802    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:25:26.420224 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:26.419897    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:25:27.202096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.202060    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr" event={"ID":"cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757","Type":"ContainerStarted","Data":"55d70ea147519b02af797a015291b5fb3dbe794ebb2a8ba0fce1f03318192f92"}
Oct 31 19:25:27.202304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.202277    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:27.203989 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.203956    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-v2zwj" event={"ID":"a0bd8286-fc8a-46e4-9c81-ca98bd386d0d","Type":"ContainerStarted","Data":"51c2f9871835f16044fbab1d1005f08aa83d549f3262dd97a5e58941b1406753"}
Oct 31 19:25:27.204129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.204025    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:27.204304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.204277    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr"
Oct 31 19:25:27.233896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.233831    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzhhr" podStartSLOduration=1.5947197819999999 podStartE2EDuration="6.233810907s" totalImagesPullingTime="4.639091125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:27.230250114 +0000 UTC m=+320.691624728" watchObservedRunningTime="2026-10-31 19:25:27.233810907 +0000 UTC m=+320.695185523"
Oct 31 19:25:27.252700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:27.252652    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-v2zwj" podStartSLOduration=2.402682903 podStartE2EDuration="7.25263494s" totalImagesPullingTime="4.849952037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:27.251140887 +0000 UTC m=+320.712515506" watchObservedRunningTime="2026-10-31 19:25:27.25263494 +0000 UTC m=+320.714009555"
Oct 31 19:25:30.226193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:30.226138    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl" event={"ID":"7f56ac52-327e-4e63-a190-346728a6e78c","Type":"ContainerStarted","Data":"bbf016b875a04649e7933e91fb8f23d891b257f250b1c668be8eb1a42e76aa32"}
Oct 31 19:25:30.226598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:30.226219    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:30.253916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:30.253864    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl" podStartSLOduration=2.898475999 podStartE2EDuration="17.253850539s" totalImagesPullingTime="14.35537454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:30.253250445 +0000 UTC m=+323.714625060" watchObservedRunningTime="2026-10-31 19:25:30.253850539 +0000 UTC m=+323.715225211"
Oct 31 19:25:38.213308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:38.213280    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-v2zwj"
Oct 31 19:25:41.237318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:41.237282    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-c79hl"
Oct 31 19:25:42.072616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:42.072584    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:25:47.044182 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:25:47.044137    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:25:56.200103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:25:56.200061    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hbjrc"
Oct 31 19:26:17.043056 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:26:17.042973    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:26:47.044304 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:26:47.044263    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:27:17.044369 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:27:17.044321    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:27:24.662112 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:24.662080    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:27:26.106871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:26.106830    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:27:26.984574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:26.984517    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:27:29.879249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:29.879210    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:27:31.607841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:31.607803    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:27:32.759902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.759868    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ggpfz"]
Oct 31 19:27:32.760452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.760434    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a1b870d2-0f39-4917-a49d-43392f3e7f90" containerName="manager"
Oct 31 19:27:32.760517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.760455    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b870d2-0f39-4917-a49d-43392f3e7f90" containerName="manager"
Oct 31 19:27:32.760596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.760566    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b870d2-0f39-4917-a49d-43392f3e7f90" containerName="manager"
Oct 31 19:27:32.763862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.763841    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.772152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.772112    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 19:27:32.772434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.772416    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5hpmb\""
Oct 31 19:27:32.772788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.772767    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 19:27:32.772964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.772945    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 19:27:32.775745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.775715    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ggpfz"]
Oct 31 19:27:32.886851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.886761    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.887059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.886962    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9td\" (UniqueName: \"kubernetes.io/projected/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-kube-api-access-cf9td\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.887128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.887114    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.887164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.887140    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.887164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.887160    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.887230 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.887181    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-home\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.987974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.987934    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.987995    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988016    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988039    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-home\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988070    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988175    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9td\" (UniqueName: \"kubernetes.io/projected/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-kube-api-access-cf9td\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988354    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988407    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-home\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.988523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.988501    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.990286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.990265    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:32.990440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:32.990424    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:33.001293 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.001268    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9td\" (UniqueName: \"kubernetes.io/projected/c104a69f-80bd-47cc-a6f0-b5e6e199e9c9-kube-api-access-cf9td\") pod \"echo-simulator-kserve-64b6f8f96c-ggpfz\" (UID: \"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:33.079099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.079016    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:33.355592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.355568    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ggpfz"]
Oct 31 19:27:33.357640 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:27:33.357612    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc104a69f_80bd_47cc_a6f0_b5e6e199e9c9.slice/crio-e114b8024882f041af6e208b837d638e206679cfd94f4ff30eb80d068ef8749e WatchSource:0}: Error finding container e114b8024882f041af6e208b837d638e206679cfd94f4ff30eb80d068ef8749e: Status 404 returned error can't find the container with id e114b8024882f041af6e208b837d638e206679cfd94f4ff30eb80d068ef8749e
Oct 31 19:27:33.359026 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.359008    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:27:33.804142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.804105    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz" event={"ID":"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9","Type":"ContainerStarted","Data":"e114b8024882f041af6e208b837d638e206679cfd94f4ff30eb80d068ef8749e"}
Oct 31 19:27:33.974788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.974742    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"]
Oct 31 19:27:33.978942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.978913    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:33.984533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.984499    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jkgv9\""
Oct 31 19:27:33.994198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:33.994170    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"]
Oct 31 19:27:34.100080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.099995    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8557080-882a-4d3e-8dde-c1f13e43564b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100064    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhzb\" (UniqueName: \"kubernetes.io/projected/f8557080-882a-4d3e-8dde-c1f13e43564b-kube-api-access-svhzb\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100098    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100128    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100172    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100195    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100238    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100274    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.100316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.100299    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201075    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201143    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201172    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201230    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8557080-882a-4d3e-8dde-c1f13e43564b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201279    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhzb\" (UniqueName: \"kubernetes.io/projected/f8557080-882a-4d3e-8dde-c1f13e43564b-kube-api-access-svhzb\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201309    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201342    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201399    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201424    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.201745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201539    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.202040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.201902    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8557080-882a-4d3e-8dde-c1f13e43564b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.203338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.202475    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.203338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.202722    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.203338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.203014    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.204464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.204442    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.204591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.204468    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.216714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.216687    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8557080-882a-4d3e-8dde-c1f13e43564b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.216824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.216775    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhzb\" (UniqueName: \"kubernetes.io/projected/f8557080-882a-4d3e-8dde-c1f13e43564b-kube-api-access-svhzb\") pod \"maas-default-gateway-openshift-default-75d7c785f-t5m2j\" (UID: \"f8557080-882a-4d3e-8dde-c1f13e43564b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.300056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.300022    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:34.597070 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.597043    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"]
Oct 31 19:27:34.599373 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:27:34.599344    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8557080_882a_4d3e_8dde_c1f13e43564b.slice/crio-b7fb817a58235a52ad41208dffd4927051dfd4a9feb7151781504dc69f5c07a0 WatchSource:0}: Error finding container b7fb817a58235a52ad41208dffd4927051dfd4a9feb7151781504dc69f5c07a0: Status 404 returned error can't find the container with id b7fb817a58235a52ad41208dffd4927051dfd4a9feb7151781504dc69f5c07a0
Oct 31 19:27:34.767640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.767606    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"]
Oct 31 19:27:34.772571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.772530    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:34.781947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.781814    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 19:27:34.784330 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.784299    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"]
Oct 31 19:27:34.818167 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.818126    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" event={"ID":"f8557080-882a-4d3e-8dde-c1f13e43564b","Type":"ContainerStarted","Data":"b7fb817a58235a52ad41208dffd4927051dfd4a9feb7151781504dc69f5c07a0"}
Oct 31 19:27:34.910557 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.910519    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:34.910764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.910592    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vzf\" (UniqueName: \"kubernetes.io/projected/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-kube-api-access-m7vzf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:34.910764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.910718    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:34.910886 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.910808    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:34.910950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.910925    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:34.911091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:34.911028    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012047 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012016    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012066    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012088    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vzf\" (UniqueName: \"kubernetes.io/projected/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-kube-api-access-m7vzf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012202    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012418 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012320    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012418 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012447    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012534    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.012715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.012682    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.014315 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.014292    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.014653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.014636    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.026035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.025979    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vzf\" (UniqueName: \"kubernetes.io/projected/3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc-kube-api-access-m7vzf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6\" (UID: \"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.094033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.093999    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:35.376075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.376050    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"]
Oct 31 19:27:35.377346 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:27:35.377320    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd8a8b1_d30c_440f_91f1_67f4f8ddd5bc.slice/crio-3d58ce17b65cac7af081e0e3619b719c791313229dccfee8649068d120eb24c6 WatchSource:0}: Error finding container 3d58ce17b65cac7af081e0e3619b719c791313229dccfee8649068d120eb24c6: Status 404 returned error can't find the container with id 3d58ce17b65cac7af081e0e3619b719c791313229dccfee8649068d120eb24c6
Oct 31 19:27:35.827666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.827567    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6" event={"ID":"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc","Type":"ContainerStarted","Data":"3d58ce17b65cac7af081e0e3619b719c791313229dccfee8649068d120eb24c6"}
Oct 31 19:27:35.830766 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.829960    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz" event={"ID":"c104a69f-80bd-47cc-a6f0-b5e6e199e9c9","Type":"ContainerStarted","Data":"2dcb77ed16eb58e7499daea4890bd8650a73b6229579f5ab3fbef845afcd60e2"}
Oct 31 19:27:35.853956 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:35.853897    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz" podStartSLOduration=2.167935847 podStartE2EDuration="3.8538821s" totalImagesPullingTime="1.685946253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:35.852771163 +0000 UTC m=+449.314145791" watchObservedRunningTime="2026-10-31 19:27:35.8538821 +0000 UTC m=+449.315256712"
Oct 31 19:27:36.080036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:36.079956    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:36.195910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:36.195878    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:27:36.836083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:36.836045    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6" event={"ID":"3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc","Type":"ContainerStarted","Data":"6b198893c14f305261660cf107030bb60188856b44f076a388493766c527f1dc"}
Oct 31 19:27:36.861696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:36.861637    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6" podStartSLOduration=1.5895580059999999 podStartE2EDuration="2.861619074s" totalImagesPullingTime="1.272061068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:36.860908388 +0000 UTC m=+450.322283003" watchObservedRunningTime="2026-10-31 19:27:36.861619074 +0000 UTC m=+450.322993689"
Oct 31 19:27:37.095212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:37.095117    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:37.876899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:37.876865    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:27:37.877307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:37.876949    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:27:37.877307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:37.876994    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:27:38.848761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:38.848723    5958 generic.go:356] "Generic (PLEG): container finished" podID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerID="24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633" exitCode=1
Oct 31 19:27:38.848970 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:38.848794    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-vdnn8" event={"ID":"4818c270-d27d-4689-9ea9-49d3b0e56579","Type":"ContainerDied","Data":"24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"}
Oct 31 19:27:38.849268 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:38.849248    5958 scope.go:122] "RemoveContainer" containerID="24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"
Oct 31 19:27:38.850656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:38.850628    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" event={"ID":"f8557080-882a-4d3e-8dde-c1f13e43564b","Type":"ContainerStarted","Data":"e7a690d58880f16b88fce41d09cf7c4cb3990689f746633d600bb4255009f7d4"}
Oct 31 19:27:39.300890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:39.300855    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:39.302464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:39.302435    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:39.302590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:39.302486    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:39.857987 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:39.857948    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-vdnn8" event={"ID":"4818c270-d27d-4689-9ea9-49d3b0e56579","Type":"ContainerStarted","Data":"281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d"}
Oct 31 19:27:39.858246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:39.858231    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:27:39.884022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:39.883957    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podStartSLOduration=3.61029043 podStartE2EDuration="6.883937277s" totalImagesPullingTime="3.273646847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:38.899805228 +0000 UTC m=+452.361179868" watchObservedRunningTime="2026-10-31 19:27:39.883937277 +0000 UTC m=+453.345311893"
Oct 31 19:27:40.300526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:40.300487    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:40.300700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:40.300555    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:41.300368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:41.300337    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:41.300767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:41.300390    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:42.301072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:42.301036    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:42.301480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:42.301102    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:43.079513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:43.079471    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:43.092017 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:43.091987    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:43.301299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:43.301262    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:43.301729 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:43.301318    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:43.888841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:43.888812    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz"
Oct 31 19:27:44.301387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:44.301291    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:44.301387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:44.301360    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:45.094708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:45.094671    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:45.113750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:45.113723    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:45.301127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:45.301091    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:45.301307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:45.301157    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 31 19:27:45.901974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:45.901937    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6"
Oct 31 19:27:46.305245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:46.305142    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:46.305570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:46.305527    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:46.306255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:46.306240    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j"
Oct 31 19:27:47.044765 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:27:47.044727    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:27:49.021749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.021710    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"]
Oct 31 19:27:49.025735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.025711    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.038798 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.038705    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"]
Oct 31 19:27:49.053522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053485    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053556    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053663    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053698    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053801    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053827    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbcx\" (UniqueName: \"kubernetes.io/projected/d2a89423-32aa-4b0c-a348-9eb35746f5b6-kube-api-access-ftbcx\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053915    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.053961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.053955    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.154884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.154848    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.154884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.154888    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.154917    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbcx\" (UniqueName: \"kubernetes.io/projected/d2a89423-32aa-4b0c-a348-9eb35746f5b6-kube-api-access-ftbcx\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.154977    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155030    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155089    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155143    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155281    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155314    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155329    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155355    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155665    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155697    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.155819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.155778    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.157435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.157414    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.157594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.157575    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.168470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.168450    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbcx\" (UniqueName: \"kubernetes.io/projected/d2a89423-32aa-4b0c-a348-9eb35746f5b6-kube-api-access-ftbcx\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.168594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.168458    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a89423-32aa-4b0c-a348-9eb35746f5b6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-dh8nw\" (UID: \"d2a89423-32aa-4b0c-a348-9eb35746f5b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.341329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.341239    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:49.622195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.622166    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"]
Oct 31 19:27:49.623585 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:27:49.623556    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a89423_32aa_4b0c_a348_9eb35746f5b6.slice/crio-bb4ea0994740cbda64bf709ece7a5c0e83cacca3f7d48c283293ea5d639a710f WatchSource:0}: Error finding container bb4ea0994740cbda64bf709ece7a5c0e83cacca3f7d48c283293ea5d639a710f: Status 404 returned error can't find the container with id bb4ea0994740cbda64bf709ece7a5c0e83cacca3f7d48c283293ea5d639a710f
Oct 31 19:27:49.626774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.626743    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:27:49.626854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.626828    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:27:49.626892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.626860    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:27:49.912350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.912269    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" event={"ID":"d2a89423-32aa-4b0c-a348-9eb35746f5b6","Type":"ContainerStarted","Data":"9f7c31c9009ac2ac7eab554e207cf54ec324800b129ee6c906375759591b6c84"}
Oct 31 19:27:49.912350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.912320    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" event={"ID":"d2a89423-32aa-4b0c-a348-9eb35746f5b6","Type":"ContainerStarted","Data":"bb4ea0994740cbda64bf709ece7a5c0e83cacca3f7d48c283293ea5d639a710f"}
Oct 31 19:27:49.940005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:49.939947    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podStartSLOduration=1.939930282 podStartE2EDuration="1.939930282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:49.93858312 +0000 UTC m=+463.399957735" watchObservedRunningTime="2026-10-31 19:27:49.939930282 +0000 UTC m=+463.401304897"
Oct 31 19:27:50.341889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:50.341798    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:50.343372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:50.343345    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:50.343471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:50.343395    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:50.865070 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:50.865039    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 19:27:51.342307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:51.342269    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:51.342755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:51.342333    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:52.342316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:52.342284    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:52.342779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:52.342338    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:53.341739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:53.341699    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:53.341927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:53.341784    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:54.342082 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:54.342043    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:54.342520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:54.342100    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:55.342034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:55.341997    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:55.342207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:55.342052    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:56.342495 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:56.342449    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-dh8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 19:27:56.342919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:56.342527    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw" podUID="d2a89423-32aa-4b0c-a348-9eb35746f5b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 19:27:58.139486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:58.139450    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:58.140016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:58.139888    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:58.140768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:58.140753    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-dh8nw"
Oct 31 19:27:59.429734 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:27:59.429695    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:28:04.056923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.056888    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"]
Oct 31 19:28:04.061102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.061070    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.078163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.078126    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"]
Oct 31 19:28:04.238458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238420    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238476    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238652    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238687    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238722    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238805    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238840    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.238924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238861    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.239033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.238933    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpf4\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.339957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.339877    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.339957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.339943    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.339965    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.339992    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340015    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340042    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340200    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340263    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpf4\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340325    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340328    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340402    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340579    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.340634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.340607    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.341067 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.341047    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.342372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.342349    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.342998 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.342974    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.357252 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.357226    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.357564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.357520    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpf4\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4\") pod \"maas-default-gateway-openshift-default-75d7c785f-tbx78\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.380016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.379985    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:04.663403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.663376    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"]
Oct 31 19:28:04.666322 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:28:04.666295    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97622e45_12bd_4731_882f_0de4f56a3aa7.slice/crio-3116cb3e62e3e29dcd800aa43f057415d79a9551e40f395643dde75e05d2748f WatchSource:0}: Error finding container 3116cb3e62e3e29dcd800aa43f057415d79a9551e40f395643dde75e05d2748f: Status 404 returned error can't find the container with id 3116cb3e62e3e29dcd800aa43f057415d79a9551e40f395643dde75e05d2748f
Oct 31 19:28:04.669411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.669378    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:28:04.669490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.669443    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:28:04.669490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:04.669473    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:28:05.008415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:05.008333    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" event={"ID":"97622e45-12bd-4731-882f-0de4f56a3aa7","Type":"ContainerStarted","Data":"9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0"}
Oct 31 19:28:05.008415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:05.008386    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" event={"ID":"97622e45-12bd-4731-882f-0de4f56a3aa7","Type":"ContainerStarted","Data":"3116cb3e62e3e29dcd800aa43f057415d79a9551e40f395643dde75e05d2748f"}
Oct 31 19:28:05.042818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:05.042761    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podStartSLOduration=1.042744796 podStartE2EDuration="1.042744796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:05.03957419 +0000 UTC m=+478.500948807" watchObservedRunningTime="2026-10-31 19:28:05.042744796 +0000 UTC m=+478.504119410"
Oct 31 19:28:05.381168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:05.381133    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:05.382637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:05.382612    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:05.382764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:05.382661    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:06.380537 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:06.380497    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:06.380746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:06.380573    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:07.381367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:07.381328    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:07.381795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:07.381389    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:08.381346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:08.381313    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:08.381515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:08.381367    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:09.234588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.234522    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"]
Oct 31 19:28:09.245530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.245495    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"]
Oct 31 19:28:09.245732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.245638    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.254166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.254135    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 19:28:09.287594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.287537    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87dh\" (UniqueName: \"kubernetes.io/projected/8eb1e811-0d07-4067-ab2f-0c75b6fd2b70-kube-api-access-m87dh\") pod \"payload-pre-processing-7b98bb678c-htgzs\" (UID: \"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.287761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.287617    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1e811-0d07-4067-ab2f-0c75b6fd2b70-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-htgzs\" (UID: \"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.380613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.380568    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:09.380806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.380632    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:09.388878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.388836    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87dh\" (UniqueName: \"kubernetes.io/projected/8eb1e811-0d07-4067-ab2f-0c75b6fd2b70-kube-api-access-m87dh\") pod \"payload-pre-processing-7b98bb678c-htgzs\" (UID: \"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.389698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.389679    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1e811-0d07-4067-ab2f-0c75b6fd2b70-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-htgzs\" (UID: \"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.390595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.390525    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eb1e811-0d07-4067-ab2f-0c75b6fd2b70-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-htgzs\" (UID: \"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.433617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.433581    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87dh\" (UniqueName: \"kubernetes.io/projected/8eb1e811-0d07-4067-ab2f-0c75b6fd2b70-kube-api-access-m87dh\") pod \"payload-pre-processing-7b98bb678c-htgzs\" (UID: \"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.560985 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.560886    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:09.928567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:09.902881    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"]
Oct 31 19:28:10.059181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:10.059138    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs" event={"ID":"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70","Type":"ContainerStarted","Data":"0054236f102d4a2ead5d092b922e380eee311a563dd5b967f82cc2a92621404d"}
Oct 31 19:28:10.381221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:10.381182    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:10.381399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:10.381237    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:11.381205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:11.381169    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:11.381757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:11.381224    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:12.380857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:12.380815    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 31 19:28:12.381061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:12.380883    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 31 19:28:13.078558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:13.078513    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs" event={"ID":"8eb1e811-0d07-4067-ab2f-0c75b6fd2b70","Type":"ContainerStarted","Data":"96c2342b08fa8210c7ad3104cd64ae7fc865e3b79a5cf82f08d4051894c9ac1c"}
Oct 31 19:28:13.079078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:13.078596    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:13.109054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:13.108964    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs" podStartSLOduration=1.8439381 podStartE2EDuration="4.108942964s" totalImagesPullingTime="2.265004864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:13.108045652 +0000 UTC m=+486.569420267" watchObservedRunningTime="2026-10-31 19:28:13.108942964 +0000 UTC m=+486.570317580"
Oct 31 19:28:14.423751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:14.423703    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:14.424342 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:14.423788    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:15.328269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:15.328230    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:15.328917 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:15.328887    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:15.330133 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:15.330102    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:28:17.044754 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:28:17.044711    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:28:19.216519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.216478    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5748ddb879-r6qth"]
Oct 31 19:28:19.220286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.220262    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.227113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.227082    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 19:28:19.229417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.229391    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 19:28:19.229578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.229467    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nmd82\""
Oct 31 19:28:19.232318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.232288    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5748ddb879-r6qth"]
Oct 31 19:28:19.297392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.297357    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6ch\" (UniqueName: \"kubernetes.io/projected/f065ea02-d1f3-4ab5-812a-4527f89650a6-kube-api-access-nt6ch\") pod \"ai-gateway-controller-5748ddb879-r6qth\" (UID: \"f065ea02-d1f3-4ab5-812a-4527f89650a6\") " pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.297392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.297395    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f065ea02-d1f3-4ab5-812a-4527f89650a6-metrics-certs\") pod \"ai-gateway-controller-5748ddb879-r6qth\" (UID: \"f065ea02-d1f3-4ab5-812a-4527f89650a6\") " pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.398938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.398904    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6ch\" (UniqueName: \"kubernetes.io/projected/f065ea02-d1f3-4ab5-812a-4527f89650a6-kube-api-access-nt6ch\") pod \"ai-gateway-controller-5748ddb879-r6qth\" (UID: \"f065ea02-d1f3-4ab5-812a-4527f89650a6\") " pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.398938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.398942    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f065ea02-d1f3-4ab5-812a-4527f89650a6-metrics-certs\") pod \"ai-gateway-controller-5748ddb879-r6qth\" (UID: \"f065ea02-d1f3-4ab5-812a-4527f89650a6\") " pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.401322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.401293    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f065ea02-d1f3-4ab5-812a-4527f89650a6-metrics-certs\") pod \"ai-gateway-controller-5748ddb879-r6qth\" (UID: \"f065ea02-d1f3-4ab5-812a-4527f89650a6\") " pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.412301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.412272    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6ch\" (UniqueName: \"kubernetes.io/projected/f065ea02-d1f3-4ab5-812a-4527f89650a6-kube-api-access-nt6ch\") pod \"ai-gateway-controller-5748ddb879-r6qth\" (UID: \"f065ea02-d1f3-4ab5-812a-4527f89650a6\") " pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.539281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.539185    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:19.815584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:19.815525    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5748ddb879-r6qth"]
Oct 31 19:28:19.817150 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:28:19.817122    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf065ea02_d1f3_4ab5_812a_4527f89650a6.slice/crio-ed6252782b8f8c70a4ea8f237baa3dc0dbd0cd1c1c71e86f541702d9eb3256c5 WatchSource:0}: Error finding container ed6252782b8f8c70a4ea8f237baa3dc0dbd0cd1c1c71e86f541702d9eb3256c5: Status 404 returned error can't find the container with id ed6252782b8f8c70a4ea8f237baa3dc0dbd0cd1c1c71e86f541702d9eb3256c5
Oct 31 19:28:20.116965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:20.116874    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth" event={"ID":"f065ea02-d1f3-4ab5-812a-4527f89650a6","Type":"ContainerStarted","Data":"ed6252782b8f8c70a4ea8f237baa3dc0dbd0cd1c1c71e86f541702d9eb3256c5"}
Oct 31 19:28:22.129010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:22.128973    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth" event={"ID":"f065ea02-d1f3-4ab5-812a-4527f89650a6","Type":"ContainerStarted","Data":"cc7c12008134257cb263d185bbad2d7dd4f1352dba299c84eb9f3bbaf40315b1"}
Oct 31 19:28:22.129413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:22.129103    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:22.150562 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:22.150486    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth" podStartSLOduration=1.589113139 podStartE2EDuration="3.150469741s" totalImagesPullingTime="1.561356602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:22.149699235 +0000 UTC m=+495.611073850" watchObservedRunningTime="2026-10-31 19:28:22.150469741 +0000 UTC m=+495.611844399"
Oct 31 19:28:24.087156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:24.087115    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-htgzs"
Oct 31 19:28:33.137316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:28:33.137284    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5748ddb879-r6qth"
Oct 31 19:28:47.048709 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:28:47.048669    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:29:07.750366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.750320    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"]
Oct 31 19:29:07.754755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.754730    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:07.767038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.767015    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5bbf75\""
Oct 31 19:29:07.784076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.784035    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"]
Oct 31 19:29:07.801196 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.801161    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"]
Oct 31 19:29:07.805811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.805789    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:07.812493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.812442    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-5bbf75-dockercfg-msd5p\""
Oct 31 19:29:07.815356 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.815319    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"]
Oct 31 19:29:07.891753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.891716    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnkch\" (UniqueName: \"kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch\") pod \"payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:07.891923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.891869    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:07.891923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.891904    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:07.892034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.891932    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jsf\" (UniqueName: \"kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf\") pod \"payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:07.992680 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.992633    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:07.992876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.992703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:07.992876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.992739    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jsf\" (UniqueName: \"kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf\") pod \"payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:07.992876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.992823    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnkch\" (UniqueName: \"kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch\") pod \"payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:07.993530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.993502    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:07.993830 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:07.993809    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:08.006596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.006469    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jsf\" (UniqueName: \"kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf\") pod \"payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:08.006939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.006911    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnkch\" (UniqueName: \"kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch\") pod \"payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:08.068795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.068745    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:08.124774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.124742    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:08.337357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.337266    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"]
Oct 31 19:29:08.342670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.342647    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.357194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.357163    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"]
Oct 31 19:29:08.363626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.363602    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-8b083a-serving-cert\""
Oct 31 19:29:08.365116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.365099    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 19:29:08.365188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.365113    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\""
Oct 31 19:29:08.374119 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:08.374087    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb977c030_04a9_4062_a654_ddbd443b3efd.slice/crio-f1a7fe07d075bcb9f02e192997778782bd2ab05d178799fe6e9efc11a44d2af2 WatchSource:0}: Error finding container f1a7fe07d075bcb9f02e192997778782bd2ab05d178799fe6e9efc11a44d2af2: Status 404 returned error can't find the container with id f1a7fe07d075bcb9f02e192997778782bd2ab05d178799fe6e9efc11a44d2af2
Oct 31 19:29:08.377577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.377537    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"]
Oct 31 19:29:08.425880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.425854    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"]
Oct 31 19:29:08.428512 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:08.428481    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c24cbec_c972_4d27_8ff5_0edcd14b20e1.slice/crio-2334779abd4fc87433f7e6e2b41c067726631ca85bb34ed66dcc601ac37ecea5 WatchSource:0}: Error finding container 2334779abd4fc87433f7e6e2b41c067726631ca85bb34ed66dcc601ac37ecea5: Status 404 returned error can't find the container with id 2334779abd4fc87433f7e6e2b41c067726631ca85bb34ed66dcc601ac37ecea5
Oct 31 19:29:08.498706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.498660    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ng2\" (UniqueName: \"kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.498861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.498720    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.498946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.498920    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.499008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.498965    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.600101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.600007    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ng2\" (UniqueName: \"kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.600101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.600055    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.600355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.600213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.600355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.600255    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.602812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.602779    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.603465 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.603443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.603621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.603490    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.637068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.637036    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ng2\" (UniqueName: \"kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2\") pod \"maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.658126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.658099    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:08.942399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.942360    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"]
Oct 31 19:29:08.948085 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.948045    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:08.950804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.950728    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"]
Oct 31 19:29:08.954299 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:08.954270    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca2385e_77ad_4075_b0df_a86eeab06863.slice/crio-9708cf3d087e0248208bf70c4fe2657361ce0f9490b426424cba0ec14fe99f22 WatchSource:0}: Error finding container 9708cf3d087e0248208bf70c4fe2657361ce0f9490b426424cba0ec14fe99f22: Status 404 returned error can't find the container with id 9708cf3d087e0248208bf70c4fe2657361ce0f9490b426424cba0ec14fe99f22
Oct 31 19:29:08.955344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.955323    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"]
Oct 31 19:29:08.955492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.955415    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:08.961772 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.961754    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-8b083a\""
Oct 31 19:29:08.973306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.973262    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"]
Oct 31 19:29:08.975631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:08.975602    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"]
Oct 31 19:29:09.004917 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.004883    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.005136 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.004950    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c25r\" (UniqueName: \"kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r\") pod \"payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.108253 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.108212    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c25r\" (UniqueName: \"kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r\") pod \"payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.108491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.108319    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.108491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.108391    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fwm\" (UniqueName: \"kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm\") pod \"payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.108491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.108464    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.109303 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.109275    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.151126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.138313    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c25r\" (UniqueName: \"kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r\") pod \"payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.209614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.209508    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.209753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.209629    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fwm\" (UniqueName: \"kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm\") pod \"payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.210448 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.210422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.226328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.226290    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fwm\" (UniqueName: \"kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm\") pod \"payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.265140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.263769    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:09.277473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.277234    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:09.386569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.385375    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd" event={"ID":"aca2385e-77ad-4075-b0df-a86eeab06863","Type":"ContainerStarted","Data":"9708cf3d087e0248208bf70c4fe2657361ce0f9490b426424cba0ec14fe99f22"}
Oct 31 19:29:09.387438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.387187    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" event={"ID":"b977c030-04a9-4062-a654-ddbd443b3efd","Type":"ContainerStarted","Data":"58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835"}
Oct 31 19:29:09.387438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.387232    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" event={"ID":"b977c030-04a9-4062-a654-ddbd443b3efd","Type":"ContainerStarted","Data":"f1a7fe07d075bcb9f02e192997778782bd2ab05d178799fe6e9efc11a44d2af2"}
Oct 31 19:29:09.387438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.387286    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:09.389384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.389350    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" event={"ID":"1c24cbec-c972-4d27-8ff5-0edcd14b20e1","Type":"ContainerStarted","Data":"43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6"}
Oct 31 19:29:09.389494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.389410    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" event={"ID":"1c24cbec-c972-4d27-8ff5-0edcd14b20e1","Type":"ContainerStarted","Data":"2334779abd4fc87433f7e6e2b41c067726631ca85bb34ed66dcc601ac37ecea5"}
Oct 31 19:29:09.390089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.390067    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:09.418259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.416608    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" podStartSLOduration=2.416591312 podStartE2EDuration="2.416591312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:09.416429696 +0000 UTC m=+542.877804312" watchObservedRunningTime="2026-10-31 19:29:09.416591312 +0000 UTC m=+542.877965927"
Oct 31 19:29:09.454274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.454221    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" podStartSLOduration=2.454208071 podStartE2EDuration="2.454208071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:09.453633298 +0000 UTC m=+542.915007913" watchObservedRunningTime="2026-10-31 19:29:09.454208071 +0000 UTC m=+542.915582700"
Oct 31 19:29:09.560449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.560366    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"]
Oct 31 19:29:09.564522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.564498    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"]
Oct 31 19:29:09.564698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.564668    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.569424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.569389    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.577302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.576790    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"]
Oct 31 19:29:09.577302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.576839    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-996c32\""
Oct 31 19:29:09.588489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.588423    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-996c32-dockercfg-7zxxp\""
Oct 31 19:29:09.590597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.590573    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"]
Oct 31 19:29:09.592581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.592530    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"]
Oct 31 19:29:09.600822 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:09.600792    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93581c47_2dc7_44c6_a6d0_78effaa42a43.slice/crio-76598ef4af12325bcc055b96b8564ae4dfcf7c787ba0e59bbe13ec3ffa8c29bc WatchSource:0}: Error finding container 76598ef4af12325bcc055b96b8564ae4dfcf7c787ba0e59bbe13ec3ffa8c29bc: Status 404 returned error can't find the container with id 76598ef4af12325bcc055b96b8564ae4dfcf7c787ba0e59bbe13ec3ffa8c29bc
Oct 31 19:29:09.602896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.602816    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"]
Oct 31 19:29:09.716920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.716884    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.717161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.717029    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4fx\" (UniqueName: \"kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx\") pod \"payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.717161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.717076    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.717445 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.717179    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrp9\" (UniqueName: \"kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9\") pod \"payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.818722 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.818638    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.818887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.818757    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4fx\" (UniqueName: \"kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx\") pod \"payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.818887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.818789    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.818887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.818868    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrp9\" (UniqueName: \"kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9\") pod \"payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.819603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.819576    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.820104 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.820081    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.833367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.833333    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrp9\" (UniqueName: \"kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9\") pod \"payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:09.833527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.833485    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4fx\" (UniqueName: \"kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx\") pod \"payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.888665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.888605    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:09.903763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:09.903537    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:10.193762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.193733    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"]
Oct 31 19:29:10.195906 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:10.195870    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fad730e_f46a_4e09_93e7_3bbf2b46aea6.slice/crio-3776670b1194d49df9862bd0f34b9ed6ddf55d684d70a4a26febb8eb4ba7062d WatchSource:0}: Error finding container 3776670b1194d49df9862bd0f34b9ed6ddf55d684d70a4a26febb8eb4ba7062d: Status 404 returned error can't find the container with id 3776670b1194d49df9862bd0f34b9ed6ddf55d684d70a4a26febb8eb4ba7062d
Oct 31 19:29:10.213199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.213147    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"]
Oct 31 19:29:10.214091 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:10.214063    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee1c945_8aaa_4d51_849f_dac9021fcbc1.slice/crio-2bf27f6d7b0761696064509ac29f37b82406418710645800d576d7e317da7571 WatchSource:0}: Error finding container 2bf27f6d7b0761696064509ac29f37b82406418710645800d576d7e317da7571: Status 404 returned error can't find the container with id 2bf27f6d7b0761696064509ac29f37b82406418710645800d576d7e317da7571
Oct 31 19:29:10.398603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.398530    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" event={"ID":"dee1c945-8aaa-4d51-849f-dac9021fcbc1","Type":"ContainerStarted","Data":"1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8"}
Oct 31 19:29:10.398794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.398630    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" event={"ID":"dee1c945-8aaa-4d51-849f-dac9021fcbc1","Type":"ContainerStarted","Data":"2bf27f6d7b0761696064509ac29f37b82406418710645800d576d7e317da7571"}
Oct 31 19:29:10.399567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.399499    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:10.401460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.401428    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" event={"ID":"8642c922-5fb7-45c8-bdcf-99dea10faa6d","Type":"ContainerStarted","Data":"42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628"}
Oct 31 19:29:10.401619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.401500    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" event={"ID":"8642c922-5fb7-45c8-bdcf-99dea10faa6d","Type":"ContainerStarted","Data":"3b6756a9b29785619b65a3c2bb08069e84b40caebf88734fe937032f4dd341ec"}
Oct 31 19:29:10.401619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.401588    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:10.403681 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.403653    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" event={"ID":"1fad730e-f46a-4e09-93e7-3bbf2b46aea6","Type":"ContainerStarted","Data":"ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38"}
Oct 31 19:29:10.403820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.403698    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" event={"ID":"1fad730e-f46a-4e09-93e7-3bbf2b46aea6","Type":"ContainerStarted","Data":"3776670b1194d49df9862bd0f34b9ed6ddf55d684d70a4a26febb8eb4ba7062d"}
Oct 31 19:29:10.403820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.403757    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:10.406722 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.406693    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" event={"ID":"93581c47-2dc7-44c6-a6d0-78effaa42a43","Type":"ContainerStarted","Data":"f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1"}
Oct 31 19:29:10.406852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.406747    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" event={"ID":"93581c47-2dc7-44c6-a6d0-78effaa42a43","Type":"ContainerStarted","Data":"76598ef4af12325bcc055b96b8564ae4dfcf7c787ba0e59bbe13ec3ffa8c29bc"}
Oct 31 19:29:10.407850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.407311    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:10.435715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.435657    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" podStartSLOduration=1.435642732 podStartE2EDuration="1.435642732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:10.434253223 +0000 UTC m=+543.895627839" watchObservedRunningTime="2026-10-31 19:29:10.435642732 +0000 UTC m=+543.897017372"
Oct 31 19:29:10.474008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.471824    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" podStartSLOduration=2.471805162 podStartE2EDuration="2.471805162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:10.471576858 +0000 UTC m=+543.932951478" watchObservedRunningTime="2026-10-31 19:29:10.471805162 +0000 UTC m=+543.933179780"
Oct 31 19:29:10.495311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.495263    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" podStartSLOduration=2.495247047 podStartE2EDuration="2.495247047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:10.494870125 +0000 UTC m=+543.956244732" watchObservedRunningTime="2026-10-31 19:29:10.495247047 +0000 UTC m=+543.956621662"
Oct 31 19:29:10.515853 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.515817    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"]
Oct 31 19:29:10.521100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.521074    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.521791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.521739    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" podStartSLOduration=1.521729792 podStartE2EDuration="1.521729792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:10.520735553 +0000 UTC m=+543.982110183" watchObservedRunningTime="2026-10-31 19:29:10.521729792 +0000 UTC m=+543.983104409"
Oct 31 19:29:10.536487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.536460    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-23843f-dockercfg-qzrgn\""
Oct 31 19:29:10.536677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.536526    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-23843f\""
Oct 31 19:29:10.538105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.538074    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"]
Oct 31 19:29:10.568840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.568802    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"]
Oct 31 19:29:10.573858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.573820    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.589429 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.589395    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"]
Oct 31 19:29:10.628754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.628719    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4882\" (UniqueName: \"kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882\") pod \"payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.628957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.628785    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljm2l\" (UniqueName: \"kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l\") pod \"payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.628957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.628878    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.628957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.628931    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.730315 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.730175    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.731709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.731658    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4882\" (UniqueName: \"kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882\") pod \"payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.731857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.731739    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljm2l\" (UniqueName: \"kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l\") pod \"payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.731857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.731850    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.731960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.731877    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.732759 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.732733    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.752224 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.752194    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljm2l\" (UniqueName: \"kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l\") pod \"payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.753486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.753451    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4882\" (UniqueName: \"kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882\") pod \"payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.839743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.839710    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:10.898997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.898966    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:10.920996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:10.920164    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:29:11.127829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.127798    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"]
Oct 31 19:29:11.130387 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:11.130351    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07738ae8_6f04_404f_80ef_06031d966bfb.slice/crio-11db9ff8c43c94ba387007eb85d9f67d4d3cc5ba07d680fcd6a8be739982e61f WatchSource:0}: Error finding container 11db9ff8c43c94ba387007eb85d9f67d4d3cc5ba07d680fcd6a8be739982e61f: Status 404 returned error can't find the container with id 11db9ff8c43c94ba387007eb85d9f67d4d3cc5ba07d680fcd6a8be739982e61f
Oct 31 19:29:11.217179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.217113    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"]
Oct 31 19:29:11.219998 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:11.219953    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65915d4_8e42_45ab_abc6_b539629821cf.slice/crio-ee518bcc84edfee9c44a6148ac9cc12a8e6b7621b2c200821d3a886d1aff4a3c WatchSource:0}: Error finding container ee518bcc84edfee9c44a6148ac9cc12a8e6b7621b2c200821d3a886d1aff4a3c: Status 404 returned error can't find the container with id ee518bcc84edfee9c44a6148ac9cc12a8e6b7621b2c200821d3a886d1aff4a3c
Oct 31 19:29:11.413934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.413849    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" event={"ID":"a65915d4-8e42-45ab-abc6-b539629821cf","Type":"ContainerStarted","Data":"5aa3c1e94be400b25464f3e90355dd67635d503df994806d5b1493cc74e405c5"}
Oct 31 19:29:11.413934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.413914    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" event={"ID":"a65915d4-8e42-45ab-abc6-b539629821cf","Type":"ContainerStarted","Data":"ee518bcc84edfee9c44a6148ac9cc12a8e6b7621b2c200821d3a886d1aff4a3c"}
Oct 31 19:29:11.414166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.414051    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:11.415770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.415746    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd" event={"ID":"aca2385e-77ad-4075-b0df-a86eeab06863","Type":"ContainerStarted","Data":"7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5"}
Oct 31 19:29:11.415875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.415804    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:11.417399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.417373    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" event={"ID":"07738ae8-6f04-404f-80ef-06031d966bfb","Type":"ContainerStarted","Data":"9598d85ed69fb7ca3f5ac276354f45e8ed46f7c638d4d073065cb9efa247f9be"}
Oct 31 19:29:11.417517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.417418    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" event={"ID":"07738ae8-6f04-404f-80ef-06031d966bfb","Type":"ContainerStarted","Data":"11db9ff8c43c94ba387007eb85d9f67d4d3cc5ba07d680fcd6a8be739982e61f"}
Oct 31 19:29:11.417517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.417483    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:11.421750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.421728    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"]
Oct 31 19:29:11.425631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.425612    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.430953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.430928    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-8730d5-dockercfg-22bzb\""
Oct 31 19:29:11.434021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.433999    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-8730d5\""
Oct 31 19:29:11.438858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.438834    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"]
Oct 31 19:29:11.439632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.439565    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" podStartSLOduration=1.439531749 podStartE2EDuration="1.439531749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:11.437512481 +0000 UTC m=+544.898887098" watchObservedRunningTime="2026-10-31 19:29:11.439531749 +0000 UTC m=+544.900906364"
Oct 31 19:29:11.487428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.487369    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" podStartSLOduration=1.487351407 podStartE2EDuration="1.487351407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:11.487038047 +0000 UTC m=+544.948412677" watchObservedRunningTime="2026-10-31 19:29:11.487351407 +0000 UTC m=+544.948726025"
Oct 31 19:29:11.512832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.512780    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd" podStartSLOduration=1.659101355 podStartE2EDuration="3.512765422s" totalImagesPullingTime="1.853664067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:11.510834537 +0000 UTC m=+544.972209144" watchObservedRunningTime="2026-10-31 19:29:11.512765422 +0000 UTC m=+544.974140038"
Oct 31 19:29:11.547390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.547344    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.547588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.547511    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvjd\" (UniqueName: \"kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd\") pod \"payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.648727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.648682    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.648938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.648779    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvjd\" (UniqueName: \"kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd\") pod \"payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.650005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.649945    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.662877 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.662842    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvjd\" (UniqueName: \"kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd\") pod \"payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.742207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.742112    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:11.870409 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.870372    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"]
Oct 31 19:29:11.876020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.875997    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:11.888116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.888088    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-aeb292\""
Oct 31 19:29:11.888275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.888167    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-aeb292-dockercfg-rtxmc\""
Oct 31 19:29:11.891005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.890977    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"]
Oct 31 19:29:11.955324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.955281    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzk96\" (UniqueName: \"kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96\") pod \"payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:11.955736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:11.955709    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:12.039443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.039421    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"]
Oct 31 19:29:12.041507 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:12.041465    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3a29c0_4c9f_4740_b379_d0d54dbb818e.slice/crio-481cbd6599a9c7405f2ab7144918f33984f39eea0c4f7a0be09182b61684bec7 WatchSource:0}: Error finding container 481cbd6599a9c7405f2ab7144918f33984f39eea0c4f7a0be09182b61684bec7: Status 404 returned error can't find the container with id 481cbd6599a9c7405f2ab7144918f33984f39eea0c4f7a0be09182b61684bec7
Oct 31 19:29:12.057878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.057846    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:12.058015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.057941    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzk96\" (UniqueName: \"kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96\") pod \"payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:12.058713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.058692    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:12.083829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.083804    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzk96\" (UniqueName: \"kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96\") pod \"payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:12.204781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.204736    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:12.424824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.424781    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" event={"ID":"3a3a29c0-4c9f-4740-b379-d0d54dbb818e","Type":"ContainerStarted","Data":"671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d"}
Oct 31 19:29:12.425362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.424857    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:12.425362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.424882    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" event={"ID":"3a3a29c0-4c9f-4740-b379-d0d54dbb818e","Type":"ContainerStarted","Data":"481cbd6599a9c7405f2ab7144918f33984f39eea0c4f7a0be09182b61684bec7"}
Oct 31 19:29:12.454460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.454398    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" podStartSLOduration=1.4543823869999999 podStartE2EDuration="1.454382387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:12.45323831 +0000 UTC m=+545.914612927" watchObservedRunningTime="2026-10-31 19:29:12.454382387 +0000 UTC m=+545.915757006"
Oct 31 19:29:12.488947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.488914    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"]
Oct 31 19:29:12.493936 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.493916    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.508019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.507931    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"]
Oct 31 19:29:12.509360 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:12.509316    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799e4486_7297_46e5_8b27_30218f000e80.slice/crio-dbbf191fa2f89b23cc1b4fb128c8fc9152912b230851b05cba3d70e5fb1eb204 WatchSource:0}: Error finding container dbbf191fa2f89b23cc1b4fb128c8fc9152912b230851b05cba3d70e5fb1eb204: Status 404 returned error can't find the container with id dbbf191fa2f89b23cc1b4fb128c8fc9152912b230851b05cba3d70e5fb1eb204
Oct 31 19:29:12.510650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.510460    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"]
Oct 31 19:29:12.568134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.568101    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.568517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.568482    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76\") pod \"payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.570082 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.570056    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-23814d\""
Oct 31 19:29:12.572152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.572128    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-23814d-dockercfg-j6tr8\""
Oct 31 19:29:12.670213 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.670177    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.670376 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.670252    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76\") pod \"payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.670976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.670950    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.683902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.683841    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76\") pod \"payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:12.810642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:12.810602    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:13.122447 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.122419    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"]
Oct 31 19:29:13.124503 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:13.124473    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ccdae5_03d1_421f_9c51_72765977005e.slice/crio-eaacb96052ad11324d2fb37bcc2a8c019d41b5893adb4271938df4bfab966f1c WatchSource:0}: Error finding container eaacb96052ad11324d2fb37bcc2a8c019d41b5893adb4271938df4bfab966f1c: Status 404 returned error can't find the container with id eaacb96052ad11324d2fb37bcc2a8c019d41b5893adb4271938df4bfab966f1c
Oct 31 19:29:13.433065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.432966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" event={"ID":"799e4486-7297-46e5-8b27-30218f000e80","Type":"ContainerStarted","Data":"845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274"}
Oct 31 19:29:13.433065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.433030    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" event={"ID":"799e4486-7297-46e5-8b27-30218f000e80","Type":"ContainerStarted","Data":"dbbf191fa2f89b23cc1b4fb128c8fc9152912b230851b05cba3d70e5fb1eb204"}
Oct 31 19:29:13.433609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.433117    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:13.434834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.434803    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" event={"ID":"87ccdae5-03d1-421f-9c51-72765977005e","Type":"ContainerStarted","Data":"222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4"}
Oct 31 19:29:13.434951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.434860    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:13.434951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.434885    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" event={"ID":"87ccdae5-03d1-421f-9c51-72765977005e","Type":"ContainerStarted","Data":"eaacb96052ad11324d2fb37bcc2a8c019d41b5893adb4271938df4bfab966f1c"}
Oct 31 19:29:13.459043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.458981    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" podStartSLOduration=2.458964939 podStartE2EDuration="2.458964939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:13.455248233 +0000 UTC m=+546.916622872" watchObservedRunningTime="2026-10-31 19:29:13.458964939 +0000 UTC m=+546.920339592"
Oct 31 19:29:13.479155 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.479120    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"]
Oct 31 19:29:13.483245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.483217    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.489825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.489776    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" podStartSLOduration=1.4897601329999999 podStartE2EDuration="1.489760133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:13.487870934 +0000 UTC m=+546.949245550" watchObservedRunningTime="2026-10-31 19:29:13.489760133 +0000 UTC m=+546.951134750"
Oct 31 19:29:13.499527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.499500    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-aeb292-serving-cert\""
Oct 31 19:29:13.503802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.503773    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"]
Oct 31 19:29:13.584475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.584429    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.584666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.584599    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.584768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.584736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplkf\" (UniqueName: \"kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.584908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.584867    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.686469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.686389    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.686469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.686447    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.686712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.686491    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplkf\" (UniqueName: \"kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.686712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.686536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.687054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.687028    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.689011 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.688986    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.689110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.689060    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.701439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.701408    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplkf\" (UniqueName: \"kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf\") pod \"maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:13.799265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:13.799228    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:14.065812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:14.065734    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:29:14.095894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:14.095864    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"]
Oct 31 19:29:14.097275 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:14.097238    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1dbc0d_2dd5_4450_9742_a8e2274e1f94.slice/crio-5ab8a62f1f570915a38de64f5b74928a9e6e339dfc401e29cad8294745c862b0 WatchSource:0}: Error finding container 5ab8a62f1f570915a38de64f5b74928a9e6e339dfc401e29cad8294745c862b0: Status 404 returned error can't find the container with id 5ab8a62f1f570915a38de64f5b74928a9e6e339dfc401e29cad8294745c862b0
Oct 31 19:29:14.442396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:14.442363    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8" event={"ID":"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94","Type":"ContainerStarted","Data":"5ab8a62f1f570915a38de64f5b74928a9e6e339dfc401e29cad8294745c862b0"}
Oct 31 19:29:15.450486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:15.450453    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8" event={"ID":"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94","Type":"ContainerStarted","Data":"c6641beba483562848c7282ab287db56d5b65ba82608df53784bf36f41617ffb"}
Oct 31 19:29:15.450897 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:15.450518    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:15.478892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:15.478831    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8" podStartSLOduration=2.083010299 podStartE2EDuration="2.478816544s" totalImagesPullingTime="395.806245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:15.477153902 +0000 UTC m=+548.938528517" watchObservedRunningTime="2026-10-31 19:29:15.478816544 +0000 UTC m=+548.940191188"
Oct 31 19:29:17.045332 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:29:17.045235    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:29:17.431640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:17.431610    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 19:29:20.411856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:20.411823    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 19:29:21.420786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:21.420755    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 19:29:21.421265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:21.421170    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 19:29:21.421265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:21.421220    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 19:29:21.463438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:21.463411    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 19:29:22.428121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:22.428087    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 19:29:22.434698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:22.434667    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 19:29:22.437100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:22.437075    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 19:29:22.438699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:22.438674    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 19:29:23.438861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:23.438829    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 19:29:24.445774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:24.445746    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 19:29:24.446264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:24.445818    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 19:29:26.911172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:26.911136    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:29:40.085816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:40.085778    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:29:47.045275 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:29:47.045229    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:29:54.232751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:54.232710    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:29:58.232749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.232708    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"]
Oct 31 19:29:58.237021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.236995    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.250421 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.250388    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.250619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.250435    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.250619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.250477    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.250742 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.250679    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.250742 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.250713    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlp9\" (UniqueName: \"kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.250845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.250790    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.252532 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.252504    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"]
Oct 31 19:29:58.252746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.252726    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"default-dockercfg-flgh4\""
Oct 31 19:29:58.252827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.252751    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"openshift-service-ca.crt\""
Oct 31 19:29:58.252827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.252807    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"kube-root-ca.crt\""
Oct 31 19:29:58.252827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.252815    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"premium-w0-23843f-kserve-self-signed-certs\""
Oct 31 19:29:58.352087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352042    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352103    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352129    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352163    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352259    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352291    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlp9\" (UniqueName: \"kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352661 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352628    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352625    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.352797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.352751    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.354406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.354380    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.354727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.354700    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.373681 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.373646    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlp9\" (UniqueName: \"kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9\") pod \"premium-w0-23843f-kserve-5c79958988-g7h82\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") " pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.549302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.549219    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:29:58.829492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:58.829459    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"]
Oct 31 19:29:58.831300 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:29:58.831271    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf54649bb_ad77_4af9_ae4c_16e9aa03f68e.slice/crio-20958a41a598e50ae5fe7919d3ba0ff5c5c84eb1a4c51fed9af5612e23e39b2c WatchSource:0}: Error finding container 20958a41a598e50ae5fe7919d3ba0ff5c5c84eb1a4c51fed9af5612e23e39b2c: Status 404 returned error can't find the container with id 20958a41a598e50ae5fe7919d3ba0ff5c5c84eb1a4c51fed9af5612e23e39b2c
Oct 31 19:29:59.766405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:59.766363    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" event={"ID":"f54649bb-ad77-4af9-ae4c-16e9aa03f68e","Type":"ContainerStarted","Data":"65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9"}
Oct 31 19:29:59.766915 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:59.766416    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" event={"ID":"f54649bb-ad77-4af9-ae4c-16e9aa03f68e","Type":"ContainerStarted","Data":"20958a41a598e50ae5fe7919d3ba0ff5c5c84eb1a4c51fed9af5612e23e39b2c"}
Oct 31 19:29:59.798727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:29:59.798658    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podStartSLOduration=1.798642624 podStartE2EDuration="1.798642624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:59.796954856 +0000 UTC m=+593.258329474" watchObservedRunningTime="2026-10-31 19:29:59.798642624 +0000 UTC m=+593.260017239"
Oct 31 19:30:00.160469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.160434    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"]
Oct 31 19:30:00.164472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.164454    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.170469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.170440    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"]
Oct 31 19:30:00.174514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.174492    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.181361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.181334    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:30:00.188309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.186012    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"]
Oct 31 19:30:00.192440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.192401    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"]
Oct 31 19:30:00.194897 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.194869    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"]
Oct 31 19:30:00.224304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.224259    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"]
Oct 31 19:30:00.224495 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.224407    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.230308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.230272    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"]
Oct 31 19:30:00.230478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.230392    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.230478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.230463    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"]
Oct 31 19:30:00.271935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.271903    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bnp6\" (UniqueName: \"kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.272108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.271947    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkqj\" (UniqueName: \"kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.272108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.271997    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6h9\" (UniqueName: \"kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.272108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.272026    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.272108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.272054    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891250-vddfm\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.272245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.272179    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.272245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.272233    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxd9\" (UniqueName: \"kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9\") pod \"maas-api-key-cleanup-29891250-vddfm\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.272324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.272293    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.373024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.372986    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.373231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373043    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxd9\" (UniqueName: \"kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9\") pod \"maas-api-key-cleanup-29891250-vddfm\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.373231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373071    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.373231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373125    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bnp6\" (UniqueName: \"kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.373231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373146    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkqj\" (UniqueName: \"kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.373231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373180    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6h9\" (UniqueName: \"kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.373231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373204    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.373673 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373257    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891250-vddfm\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.373839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373814    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.373934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.373907    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.374024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.374005    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.374059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.374044    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891250-vddfm\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.388854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.388819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6h9\" (UniqueName: \"kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.389004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.388878    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bnp6\" (UniqueName: \"kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.389004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.388963    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkqj\" (UniqueName: \"kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.389207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.389188    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxd9\" (UniqueName: \"kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9\") pod \"maas-api-key-cleanup-29891250-vddfm\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.479370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.479281    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:00.500216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.497757    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:00.543216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.543180    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:00.550391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.550365    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:30:00.556434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.554641    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:00.779466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.779444    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"]
Oct 31 19:30:00.781353 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:00.781326    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78d82c8_c8b6_4719_b327_6e4507e9d6de.slice/crio-9ccee38e6555a61688df4eeb276d49b8bc319fb3c8ce00d8d833df840dd3c8e1 WatchSource:0}: Error finding container 9ccee38e6555a61688df4eeb276d49b8bc319fb3c8ce00d8d833df840dd3c8e1: Status 404 returned error can't find the container with id 9ccee38e6555a61688df4eeb276d49b8bc319fb3c8ce00d8d833df840dd3c8e1
Oct 31 19:30:00.784501 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:00.784476    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37407655_f69c_46c3_8c09_2d24684cf26e.slice/crio-def09d76a7161e5582611459932d3629005751ca07a3b66d20573ca4184599a2 WatchSource:0}: Error finding container def09d76a7161e5582611459932d3629005751ca07a3b66d20573ca4184599a2: Status 404 returned error can't find the container with id def09d76a7161e5582611459932d3629005751ca07a3b66d20573ca4184599a2
Oct 31 19:30:00.784635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.784509    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"]
Oct 31 19:30:00.840373 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.840346    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"]
Oct 31 19:30:00.842321 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:00.842294    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9101d611_3a32_4ea0_b530_c58ea998821f.slice/crio-2ffd08dc295e6971caffba10d0eadb8681c326dde576940561a07d1af3a232fc WatchSource:0}: Error finding container 2ffd08dc295e6971caffba10d0eadb8681c326dde576940561a07d1af3a232fc: Status 404 returned error can't find the container with id 2ffd08dc295e6971caffba10d0eadb8681c326dde576940561a07d1af3a232fc
Oct 31 19:30:00.866174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:00.866148    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"]
Oct 31 19:30:00.867097 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:00.867074    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35c9173_da5b_444e_ba39_18560aa5f2be.slice/crio-7d0a2be602ebef2ff0517ba3b31a94292ed1564a3082938dd18321627566aba7 WatchSource:0}: Error finding container 7d0a2be602ebef2ff0517ba3b31a94292ed1564a3082938dd18321627566aba7: Status 404 returned error can't find the container with id 7d0a2be602ebef2ff0517ba3b31a94292ed1564a3082938dd18321627566aba7
Oct 31 19:30:01.785044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:01.785001    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerStarted","Data":"7d0a2be602ebef2ff0517ba3b31a94292ed1564a3082938dd18321627566aba7"}
Oct 31 19:30:01.786369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:01.786343    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" event={"ID":"37407655-f69c-46c3-8c09-2d24684cf26e","Type":"ContainerStarted","Data":"def09d76a7161e5582611459932d3629005751ca07a3b66d20573ca4184599a2"}
Oct 31 19:30:01.787584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:01.787559    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" event={"ID":"e78d82c8-c8b6-4719-b327-6e4507e9d6de","Type":"ContainerStarted","Data":"9ccee38e6555a61688df4eeb276d49b8bc319fb3c8ce00d8d833df840dd3c8e1"}
Oct 31 19:30:01.788731 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:01.788705    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" event={"ID":"9101d611-3a32-4ea0-b530-c58ea998821f","Type":"ContainerStarted","Data":"2ffd08dc295e6971caffba10d0eadb8681c326dde576940561a07d1af3a232fc"}
Oct 31 19:30:05.837686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.837651    5958 generic.go:356] "Generic (PLEG): container finished" podID="9101d611-3a32-4ea0-b530-c58ea998821f" containerID="b1b239887e924eb5566e3f8875233a664898a38cc2ab738ded647533c0164b3d" exitCode=60
Oct 31 19:30:05.838152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.837726    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" event={"ID":"9101d611-3a32-4ea0-b530-c58ea998821f","Type":"ContainerDied","Data":"b1b239887e924eb5566e3f8875233a664898a38cc2ab738ded647533c0164b3d"}
Oct 31 19:30:05.838152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.838037    5958 scope.go:122] "RemoveContainer" containerID="b1b239887e924eb5566e3f8875233a664898a38cc2ab738ded647533c0164b3d"
Oct 31 19:30:05.839503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.839475    5958 generic.go:356] "Generic (PLEG): container finished" podID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerID="be045d8b419e7d7724a4c1d2efe480dba9cbb47e83788f1d1c11376554102301" exitCode=60
Oct 31 19:30:05.839657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.839568    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerDied","Data":"be045d8b419e7d7724a4c1d2efe480dba9cbb47e83788f1d1c11376554102301"}
Oct 31 19:30:05.839833 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.839816    5958 scope.go:122] "RemoveContainer" containerID="be045d8b419e7d7724a4c1d2efe480dba9cbb47e83788f1d1c11376554102301"
Oct 31 19:30:05.841297 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.841269    5958 generic.go:356] "Generic (PLEG): container finished" podID="37407655-f69c-46c3-8c09-2d24684cf26e" containerID="fef4f45dbd878dc975f7448ccc9d8f9aa4b6e4fa2a14aa173a61bbe367395c0d" exitCode=60
Oct 31 19:30:05.841402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.841389    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" event={"ID":"37407655-f69c-46c3-8c09-2d24684cf26e","Type":"ContainerDied","Data":"fef4f45dbd878dc975f7448ccc9d8f9aa4b6e4fa2a14aa173a61bbe367395c0d"}
Oct 31 19:30:05.841625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.841609    5958 scope.go:122] "RemoveContainer" containerID="fef4f45dbd878dc975f7448ccc9d8f9aa4b6e4fa2a14aa173a61bbe367395c0d"
Oct 31 19:30:05.843202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.843177    5958 generic.go:356] "Generic (PLEG): container finished" podID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerID="5cc569d308b576a267dd7085896984115c9707fce5cef9f0382235fb1f785cdf" exitCode=60
Oct 31 19:30:05.843280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.843257    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" event={"ID":"e78d82c8-c8b6-4719-b327-6e4507e9d6de","Type":"ContainerDied","Data":"5cc569d308b576a267dd7085896984115c9707fce5cef9f0382235fb1f785cdf"}
Oct 31 19:30:05.843501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:05.843454    5958 scope.go:122] "RemoveContainer" containerID="5cc569d308b576a267dd7085896984115c9707fce5cef9f0382235fb1f785cdf"
Oct 31 19:30:06.852921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.852875    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" event={"ID":"9101d611-3a32-4ea0-b530-c58ea998821f","Type":"ContainerStarted","Data":"c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c"}
Oct 31 19:30:06.854946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.854913    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerStarted","Data":"4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613"}
Oct 31 19:30:06.856753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.856725    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" event={"ID":"37407655-f69c-46c3-8c09-2d24684cf26e","Type":"ContainerStarted","Data":"8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2"}
Oct 31 19:30:06.858553 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.858520    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" event={"ID":"e78d82c8-c8b6-4719-b327-6e4507e9d6de","Type":"ContainerStarted","Data":"57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c"}
Oct 31 19:30:06.874070 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.874018    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" podStartSLOduration=2.977698615 podStartE2EDuration="6.874004345s" totalImagesPullingTime="3.89630573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:06.87282523 +0000 UTC m=+600.334199845" watchObservedRunningTime="2026-10-31 19:30:06.874004345 +0000 UTC m=+600.335378959"
Oct 31 19:30:06.897625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.897572    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" podStartSLOduration=3.033299164 podStartE2EDuration="6.897532667s" totalImagesPullingTime="3.864233503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:06.897332279 +0000 UTC m=+600.358706896" watchObservedRunningTime="2026-10-31 19:30:06.897532667 +0000 UTC m=+600.358907279"
Oct 31 19:30:06.921480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.921430    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" podStartSLOduration=2.948132244 podStartE2EDuration="6.921415639s" totalImagesPullingTime="3.973283395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:06.920775534 +0000 UTC m=+600.382150149" watchObservedRunningTime="2026-10-31 19:30:06.921415639 +0000 UTC m=+600.382790253"
Oct 31 19:30:06.940560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:06.940494    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" podStartSLOduration=3.069770742 podStartE2EDuration="6.9404805s" totalImagesPullingTime="3.870709758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:06.939854549 +0000 UTC m=+600.401229165" watchObservedRunningTime="2026-10-31 19:30:06.9404805 +0000 UTC m=+600.401855115"
Oct 31 19:30:07.866920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.866883    5958 generic.go:356] "Generic (PLEG): container finished" podID="9101d611-3a32-4ea0-b530-c58ea998821f" containerID="c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c" exitCode=60
Oct 31 19:30:07.867402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.866976    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" event={"ID":"9101d611-3a32-4ea0-b530-c58ea998821f","Type":"ContainerDied","Data":"c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c"}
Oct 31 19:30:07.867402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.867044    5958 scope.go:122] "RemoveContainer" containerID="b1b239887e924eb5566e3f8875233a664898a38cc2ab738ded647533c0164b3d"
Oct 31 19:30:07.867402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.867260    5958 scope.go:122] "RemoveContainer" containerID="c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c"
Oct 31 19:30:07.867607 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:07.867581    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5_odh-ai-gateway-infra(9101d611-3a32-4ea0-b530-c58ea998821f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" podUID="9101d611-3a32-4ea0-b530-c58ea998821f"
Oct 31 19:30:07.869295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.869272    5958 generic.go:356] "Generic (PLEG): container finished" podID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerID="4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613" exitCode=60
Oct 31 19:30:07.869396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.869337    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerDied","Data":"4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613"}
Oct 31 19:30:07.869706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.869687    5958 scope.go:122] "RemoveContainer" containerID="4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613"
Oct 31 19:30:07.870025 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:07.869977    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w_odh-ai-gateway-infra(d35c9173-da5b-444e-ba39-18560aa5f2be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be"
Oct 31 19:30:07.871350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.871317    5958 generic.go:356] "Generic (PLEG): container finished" podID="37407655-f69c-46c3-8c09-2d24684cf26e" containerID="8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2" exitCode=60
Oct 31 19:30:07.871452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.871401    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" event={"ID":"37407655-f69c-46c3-8c09-2d24684cf26e","Type":"ContainerDied","Data":"8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2"}
Oct 31 19:30:07.871701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.871683    5958 scope.go:122] "RemoveContainer" containerID="8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2"
Oct 31 19:30:07.871973 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:07.871952    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891250-vddfm_odh-ai-gateway-infra(37407655-f69c-46c3-8c09-2d24684cf26e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" podUID="37407655-f69c-46c3-8c09-2d24684cf26e"
Oct 31 19:30:07.873506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.873480    5958 generic.go:356] "Generic (PLEG): container finished" podID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerID="57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c" exitCode=60
Oct 31 19:30:07.873626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.873587    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" event={"ID":"e78d82c8-c8b6-4719-b327-6e4507e9d6de","Type":"ContainerDied","Data":"57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c"}
Oct 31 19:30:07.873854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.873842    5958 scope.go:122] "RemoveContainer" containerID="57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c"
Oct 31 19:30:07.874081 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:07.874065    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6_odh-ai-gateway-infra(e78d82c8-c8b6-4719-b327-6e4507e9d6de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de"
Oct 31 19:30:07.880715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.880693    5958 scope.go:122] "RemoveContainer" containerID="be045d8b419e7d7724a4c1d2efe480dba9cbb47e83788f1d1c11376554102301"
Oct 31 19:30:07.914725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.914699    5958 scope.go:122] "RemoveContainer" containerID="fef4f45dbd878dc975f7448ccc9d8f9aa4b6e4fa2a14aa173a61bbe367395c0d"
Oct 31 19:30:07.928843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:07.928819    5958 scope.go:122] "RemoveContainer" containerID="5cc569d308b576a267dd7085896984115c9707fce5cef9f0382235fb1f785cdf"
Oct 31 19:30:08.550172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.550133    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:30:08.563190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.563162    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:30:08.881230 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.881200    5958 scope.go:122] "RemoveContainer" containerID="8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2"
Oct 31 19:30:08.881717 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:08.881513    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891250-vddfm_odh-ai-gateway-infra(37407655-f69c-46c3-8c09-2d24684cf26e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" podUID="37407655-f69c-46c3-8c09-2d24684cf26e"
Oct 31 19:30:08.883048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.883025    5958 scope.go:122] "RemoveContainer" containerID="57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c"
Oct 31 19:30:08.883314 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:08.883295    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6_odh-ai-gateway-infra(e78d82c8-c8b6-4719-b327-6e4507e9d6de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de"
Oct 31 19:30:08.884762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.884744    5958 scope.go:122] "RemoveContainer" containerID="c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c"
Oct 31 19:30:08.885026 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:08.885006    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5_odh-ai-gateway-infra(9101d611-3a32-4ea0-b530-c58ea998821f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" podUID="9101d611-3a32-4ea0-b530-c58ea998821f"
Oct 31 19:30:08.886490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.886470    5958 scope.go:122] "RemoveContainer" containerID="4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613"
Oct 31 19:30:08.886716 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:08.886690    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w_odh-ai-gateway-infra(d35c9173-da5b-444e-ba39-18560aa5f2be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be"
Oct 31 19:30:08.897831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:08.897806    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:30:13.412245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:13.412197    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:30:17.052495 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:17.052449    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:30:17.784636 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.784606    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"]
Oct 31 19:30:17.790294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.790271    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.800028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.800000    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"kube-root-ca.crt\""
Oct 31 19:30:17.800226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.800073    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"openshift-service-ca.crt\""
Oct 31 19:30:17.801273 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.801247    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"]
Oct 31 19:30:17.803470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.803447    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"premium-w3-23814d-kserve-self-signed-certs\""
Oct 31 19:30:17.803470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.803461    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"default-dockercfg-z7g59\""
Oct 31 19:30:17.876520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.876487    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.876689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.876559    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.876689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.876677    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.876765 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.876706    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.876802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.876779    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q48n\" (UniqueName: \"kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.876883 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.876866    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978017 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.977984    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978045    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978086    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978125    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978140    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978179    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q48n\" (UniqueName: \"kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978538    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978564    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.978704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.978636    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.980609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.980588    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.980728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.980645    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:17.992865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:17.992833    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q48n\" (UniqueName: \"kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n\") pod \"premium-w3-23814d-kserve-77cccbf599-nwt6g\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") " pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:18.105231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.105142    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:18.396192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.396168    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"]
Oct 31 19:30:18.398103 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:18.398076    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b812a42_fb15_427c_a378_e852f22fc881.slice/crio-336d95e71854cf866f0d1ec71d06b343c76c5235880797435d83a417fa75552a WatchSource:0}: Error finding container 336d95e71854cf866f0d1ec71d06b343c76c5235880797435d83a417fa75552a: Status 404 returned error can't find the container with id 336d95e71854cf866f0d1ec71d06b343c76c5235880797435d83a417fa75552a
Oct 31 19:30:18.656724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.656642    5958 scope.go:122] "RemoveContainer" containerID="4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613"
Oct 31 19:30:18.961117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.961076    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" event={"ID":"5b812a42-fb15-427c-a378-e852f22fc881","Type":"ContainerStarted","Data":"24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272"}
Oct 31 19:30:18.961296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.961129    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" event={"ID":"5b812a42-fb15-427c-a378-e852f22fc881","Type":"ContainerStarted","Data":"336d95e71854cf866f0d1ec71d06b343c76c5235880797435d83a417fa75552a"}
Oct 31 19:30:18.962986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.962957    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerStarted","Data":"2f744bca1792f14c2b34e44c4e0760ebca2d0549a26516486e8038528a1c10d4"}
Oct 31 19:30:18.985002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:18.984953    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podStartSLOduration=1.98494028 podStartE2EDuration="1.98494028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:18.982038061 +0000 UTC m=+612.443412681" watchObservedRunningTime="2026-10-31 19:30:18.98494028 +0000 UTC m=+612.446314925"
Oct 31 19:30:19.123089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.113334    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:19.656286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.656255    5958 scope.go:122] "RemoveContainer" containerID="c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c"
Oct 31 19:30:19.656462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.656361    5958 scope.go:122] "RemoveContainer" containerID="57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c"
Oct 31 19:30:19.970878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.970797    5958 generic.go:356] "Generic (PLEG): container finished" podID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerID="2f744bca1792f14c2b34e44c4e0760ebca2d0549a26516486e8038528a1c10d4" exitCode=60
Oct 31 19:30:19.971021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.970871    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerDied","Data":"2f744bca1792f14c2b34e44c4e0760ebca2d0549a26516486e8038528a1c10d4"}
Oct 31 19:30:19.971021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.970934    5958 scope.go:122] "RemoveContainer" containerID="4d9503ca92cede6df385a03419997d57696bf2425f50cc54975182301a42f613"
Oct 31 19:30:19.971473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:19.971450    5958 scope.go:122] "RemoveContainer" containerID="2f744bca1792f14c2b34e44c4e0760ebca2d0549a26516486e8038528a1c10d4"
Oct 31 19:30:19.971765 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:19.971741    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w_odh-ai-gateway-infra(d35c9173-da5b-444e-ba39-18560aa5f2be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be"
Oct 31 19:30:20.038340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.038295    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"]
Oct 31 19:30:20.873350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.873299    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"]
Oct 31 19:30:20.877601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.877571    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.890290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.888510    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"kube-root-ca.crt\""
Oct 31 19:30:20.890290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.888750    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"default-dockercfg-b5gbj\""
Oct 31 19:30:20.891336 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.891311    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"openshift-service-ca.crt\""
Oct 31 19:30:20.891534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.891397    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"premium-w6-8730d5-kserve-self-signed-certs\""
Oct 31 19:30:20.891534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.891468    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"]
Oct 31 19:30:20.922953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.922907    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.923147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.923036    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dfc\" (UniqueName: \"kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.923147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.923081    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.923147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.923104    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.923312 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.923262    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.923312 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.923303    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:20.978366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.978333    5958 generic.go:356] "Generic (PLEG): container finished" podID="9101d611-3a32-4ea0-b530-c58ea998821f" containerID="b9379388c99a49367b35ae8deadd5266f4981d834bd09f534911135102ec934b" exitCode=60
Oct 31 19:30:20.978573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.978409    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" event={"ID":"9101d611-3a32-4ea0-b530-c58ea998821f","Type":"ContainerDied","Data":"b9379388c99a49367b35ae8deadd5266f4981d834bd09f534911135102ec934b"}
Oct 31 19:30:20.978573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.978477    5958 scope.go:122] "RemoveContainer" containerID="c3cedb03e6084a845f8dcfc3699f90f7280ac8454386904f2a3df519fe417c4c"
Oct 31 19:30:20.978933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.978915    5958 scope.go:122] "RemoveContainer" containerID="b9379388c99a49367b35ae8deadd5266f4981d834bd09f534911135102ec934b"
Oct 31 19:30:20.979258 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:20.979227    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5_odh-ai-gateway-infra(9101d611-3a32-4ea0-b530-c58ea998821f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" podUID="9101d611-3a32-4ea0-b530-c58ea998821f"
Oct 31 19:30:20.982128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.982105    5958 generic.go:356] "Generic (PLEG): container finished" podID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerID="0dd9029b01954cf2373627274b66904435279284082adccc982c2d930cbe655b" exitCode=60
Oct 31 19:30:20.982268 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.982172    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" event={"ID":"e78d82c8-c8b6-4719-b327-6e4507e9d6de","Type":"ContainerDied","Data":"0dd9029b01954cf2373627274b66904435279284082adccc982c2d930cbe655b"}
Oct 31 19:30:20.982675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.982654    5958 scope.go:122] "RemoveContainer" containerID="0dd9029b01954cf2373627274b66904435279284082adccc982c2d930cbe655b"
Oct 31 19:30:20.982914 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:20.982895    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6_odh-ai-gateway-infra(e78d82c8-c8b6-4719-b327-6e4507e9d6de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de"
Oct 31 19:30:20.991260 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:20.991240    5958 scope.go:122] "RemoveContainer" containerID="57be7841ceec425bbc73f4d9923195fa440b016e881578e2a74e22caf12e215c"
Oct 31 19:30:21.024363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024329    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.024573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024404    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dfc\" (UniqueName: \"kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.024573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024440    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.024573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024467    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024601    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024627    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.024820    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.025034    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.025118    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.027182    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.030281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.027590    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.047701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.046819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dfc\" (UniqueName: \"kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc\") pod \"premium-w6-8730d5-kserve-5f68b56bc9-2cljd\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") " pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.156477 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.156450    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:21.195346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.195314    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:21.227925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.227893    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca\") pod \"d35c9173-da5b-444e-ba39-18560aa5f2be\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") "
Oct 31 19:30:21.228103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.227968    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9\" (UniqueName: \"kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9\") pod \"d35c9173-da5b-444e-ba39-18560aa5f2be\" (UID: \"d35c9173-da5b-444e-ba39-18560aa5f2be\") "
Oct 31 19:30:21.228337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.228307    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca" pod "d35c9173-da5b-444e-ba39-18560aa5f2be" (UID: "d35c9173-da5b-444e-ba39-18560aa5f2be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:21.230263 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.230231    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9" pod "d35c9173-da5b-444e-ba39-18560aa5f2be" (UID: "d35c9173-da5b-444e-ba39-18560aa5f2be"). InnerVolumeSpecName "kube-api-access-xh6h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:21.330131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.330089    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c9173-da5b-444e-ba39-18560aa5f2be-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.330131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.330135    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6h9\" (UniqueName: \"kubernetes.io/projected/d35c9173-da5b-444e-ba39-18560aa5f2be-kube-api-access-xh6h9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.476604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.476576    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"]
Oct 31 19:30:21.477751 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:21.477716    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a15b93d_e8a6_4a8e_ba3a_fe9ced40c4e0.slice/crio-fa25a03532e3c4e6b67b9651b5a8e772e7767a1b35acbfd94a1d149c3bbe34d1 WatchSource:0}: Error finding container fa25a03532e3c4e6b67b9651b5a8e772e7767a1b35acbfd94a1d149c3bbe34d1: Status 404 returned error can't find the container with id fa25a03532e3c4e6b67b9651b5a8e772e7767a1b35acbfd94a1d149c3bbe34d1
Oct 31 19:30:21.656753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.656720    5958 scope.go:122] "RemoveContainer" containerID="8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2"
Oct 31 19:30:21.993125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.993087    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" event={"ID":"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0","Type":"ContainerStarted","Data":"6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811"}
Oct 31 19:30:21.993125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.993130    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" event={"ID":"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0","Type":"ContainerStarted","Data":"fa25a03532e3c4e6b67b9651b5a8e772e7767a1b35acbfd94a1d149c3bbe34d1"}
Oct 31 19:30:21.994348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.994327    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w" event={"ID":"d35c9173-da5b-444e-ba39-18560aa5f2be","Type":"ContainerDied","Data":"7d0a2be602ebef2ff0517ba3b31a94292ed1564a3082938dd18321627566aba7"}
Oct 31 19:30:21.994464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.994381    5958 scope.go:122] "RemoveContainer" containerID="2f744bca1792f14c2b34e44c4e0760ebca2d0549a26516486e8038528a1c10d4"
Oct 31 19:30:21.994464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:21.994331    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"
Oct 31 19:30:22.026110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.026084    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"]
Oct 31 19:30:22.027475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.027425    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podStartSLOduration=2.027411635 podStartE2EDuration="2.027411635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:22.025051581 +0000 UTC m=+615.486426199" watchObservedRunningTime="2026-10-31 19:30:22.027411635 +0000 UTC m=+615.488786255"
Oct 31 19:30:22.056330 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.056296    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"]
Oct 31 19:30:22.066615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.066578    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"]
Oct 31 19:30:22.080475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.080380    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891250-ptt4w"]
Oct 31 19:30:22.204469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.198898    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:22.227145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.227115    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:22.298844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.298822    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:22.341092 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341043    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca\") pod \"9101d611-3a32-4ea0-b530-c58ea998821f\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") "
Oct 31 19:30:22.341254 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341178    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj\" (UniqueName: \"kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj\") pod \"9101d611-3a32-4ea0-b530-c58ea998821f\" (UID: \"9101d611-3a32-4ea0-b530-c58ea998821f\") "
Oct 31 19:30:22.341297 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341257    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6\" (UniqueName: \"kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6\") pod \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") "
Oct 31 19:30:22.341334 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341315    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca\") pod \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\" (UID: \"e78d82c8-c8b6-4719-b327-6e4507e9d6de\") "
Oct 31 19:30:22.341413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341389    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca" pod "9101d611-3a32-4ea0-b530-c58ea998821f" (UID: "9101d611-3a32-4ea0-b530-c58ea998821f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:22.341832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341801    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9101d611-3a32-4ea0-b530-c58ea998821f-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:22.341961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.341938    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca" pod "e78d82c8-c8b6-4719-b327-6e4507e9d6de" (UID: "e78d82c8-c8b6-4719-b327-6e4507e9d6de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:22.343387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.343358    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj" pod "9101d611-3a32-4ea0-b530-c58ea998821f" (UID: "9101d611-3a32-4ea0-b530-c58ea998821f"). InnerVolumeSpecName "kube-api-access-lpkqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:22.343492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.343473    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6" pod "e78d82c8-c8b6-4719-b327-6e4507e9d6de" (UID: "e78d82c8-c8b6-4719-b327-6e4507e9d6de"). InnerVolumeSpecName "kube-api-access-8bnp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:22.442851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.442776    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpkqj\" (UniqueName: \"kubernetes.io/projected/9101d611-3a32-4ea0-b530-c58ea998821f-kube-api-access-lpkqj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:22.442851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.442804    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bnp6\" (UniqueName: \"kubernetes.io/projected/e78d82c8-c8b6-4719-b327-6e4507e9d6de-kube-api-access-8bnp6\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:22.442851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.442814    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78d82c8-c8b6-4719-b327-6e4507e9d6de-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:22.664613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:22.664580    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" path="/var/lib/kubelet/pods/d35c9173-da5b-444e-ba39-18560aa5f2be/volumes"
Oct 31 19:30:23.002087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.002058    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"
Oct 31 19:30:23.002516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.002105    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5" event={"ID":"9101d611-3a32-4ea0-b530-c58ea998821f","Type":"ContainerDied","Data":"2ffd08dc295e6971caffba10d0eadb8681c326dde576940561a07d1af3a232fc"}
Oct 31 19:30:23.002516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.002173    5958 scope.go:122] "RemoveContainer" containerID="b9379388c99a49367b35ae8deadd5266f4981d834bd09f534911135102ec934b"
Oct 31 19:30:23.006364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.006334    5958 generic.go:356] "Generic (PLEG): container finished" podID="37407655-f69c-46c3-8c09-2d24684cf26e" containerID="ad68ccba368970b0143a99eaf8434697b284402619c72871e584e5a1599d816d" exitCode=60
Oct 31 19:30:23.006494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.006409    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" event={"ID":"37407655-f69c-46c3-8c09-2d24684cf26e","Type":"ContainerDied","Data":"ad68ccba368970b0143a99eaf8434697b284402619c72871e584e5a1599d816d"}
Oct 31 19:30:23.006916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.006892    5958 scope.go:122] "RemoveContainer" containerID="ad68ccba368970b0143a99eaf8434697b284402619c72871e584e5a1599d816d"
Oct 31 19:30:23.007247 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:23.007218    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891250-vddfm_odh-ai-gateway-infra(37407655-f69c-46c3-8c09-2d24684cf26e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" podUID="37407655-f69c-46c3-8c09-2d24684cf26e"
Oct 31 19:30:23.008427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.008398    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"
Oct 31 19:30:23.008730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.008705    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6" event={"ID":"e78d82c8-c8b6-4719-b327-6e4507e9d6de","Type":"ContainerDied","Data":"9ccee38e6555a61688df4eeb276d49b8bc319fb3c8ce00d8d833df840dd3c8e1"}
Oct 31 19:30:23.015318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.015295    5958 scope.go:122] "RemoveContainer" containerID="8cd38b25fe28123a127e9bbe6fb4bd2efe67e7a065e4cae859eba2f428d252b2"
Oct 31 19:30:23.033497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.033475    5958 scope.go:122] "RemoveContainer" containerID="0dd9029b01954cf2373627274b66904435279284082adccc982c2d930cbe655b"
Oct 31 19:30:23.048942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.048916    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"]
Oct 31 19:30:23.065416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.065393    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891250-lmxn5"]
Oct 31 19:30:23.100431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.100389    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"]
Oct 31 19:30:23.108843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:23.108813    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891250-nx8n6"]
Oct 31 19:30:24.124701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.124662    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"]
Oct 31 19:30:24.324381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.324356    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:24.481318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.481230    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9\" (UniqueName: \"kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9\") pod \"37407655-f69c-46c3-8c09-2d24684cf26e\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") "
Oct 31 19:30:24.481318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.481314    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca\") pod \"37407655-f69c-46c3-8c09-2d24684cf26e\" (UID: \"37407655-f69c-46c3-8c09-2d24684cf26e\") "
Oct 31 19:30:24.481687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.481664    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca" pod "37407655-f69c-46c3-8c09-2d24684cf26e" (UID: "37407655-f69c-46c3-8c09-2d24684cf26e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:24.483407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.483373    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9" pod "37407655-f69c-46c3-8c09-2d24684cf26e" (UID: "37407655-f69c-46c3-8c09-2d24684cf26e"). InnerVolumeSpecName "kube-api-access-2kxd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:24.582342 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.582307    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37407655-f69c-46c3-8c09-2d24684cf26e-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:24.582342 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.582338    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxd9\" (UniqueName: \"kubernetes.io/projected/37407655-f69c-46c3-8c09-2d24684cf26e-kube-api-access-2kxd9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:24.663177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.663132    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" path="/var/lib/kubelet/pods/9101d611-3a32-4ea0-b530-c58ea998821f/volumes"
Oct 31 19:30:24.663675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:24.663653    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" path="/var/lib/kubelet/pods/e78d82c8-c8b6-4719-b327-6e4507e9d6de/volumes"
Oct 31 19:30:25.031088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:25.031052    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"
Oct 31 19:30:25.031305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:25.031101    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm" event={"ID":"37407655-f69c-46c3-8c09-2d24684cf26e","Type":"ContainerDied","Data":"def09d76a7161e5582611459932d3629005751ca07a3b66d20573ca4184599a2"}
Oct 31 19:30:25.031305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:25.031168    5958 scope.go:122] "RemoveContainer" containerID="ad68ccba368970b0143a99eaf8434697b284402619c72871e584e5a1599d816d"
Oct 31 19:30:25.052696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:25.052661    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"]
Oct 31 19:30:25.058192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:25.058161    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891250-vddfm"]
Oct 31 19:30:26.680075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:26.680038    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" path="/var/lib/kubelet/pods/37407655-f69c-46c3-8c09-2d24684cf26e/volumes"
Oct 31 19:30:28.105579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:28.105525    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:28.118505 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:28.118477    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:29.068793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:29.068762    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:30:30.808369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:30.808333    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:30:31.196167 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:31.196120    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:31.231305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:31.231271    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:32.086220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:32.086184    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:30:41.323615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.323577    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"]
Oct 31 19:30:41.324475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324450    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324475    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324491    5958 cpu_manager.go:414] "containerMap: removing container" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324498    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324514    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324522    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324535    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324566    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324583    5958 cpu_manager.go:414] "containerMap: removing container" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:30:41.324604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324590    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324612    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324620    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324634    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324640    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324661    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324668    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324690    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324698    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324867    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324878    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324889    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324898    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324911    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324920    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324932    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:30:41.325042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.324945    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:30:41.332818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.332790    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.335656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.335627    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"]
Oct 31 19:30:41.337984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.337959    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:30:41.340044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.340023    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:30:41.402117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.402082    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.402281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.402137    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.402402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.402381    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctw7k\" (UniqueName: \"kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.503453 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.503413    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.503683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.503464    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.503683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.503614    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctw7k\" (UniqueName: \"kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.504232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.504207    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.505864 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.505838    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.517892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.517861    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctw7k\" (UniqueName: \"kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k\") pod \"maas-api-revoke-sub-820a1ecc-xvjkw\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.654764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.654731    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:41.930848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:41.930820    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"]
Oct 31 19:30:41.931980 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:30:41.931942    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9ed52f_9c85_4647_a080_a8392a1c982b.slice/crio-73f2b4181f8514c1d61ef7f1520cfabcf32edd10fe4ff9a26b5140be078fda42 WatchSource:0}: Error finding container 73f2b4181f8514c1d61ef7f1520cfabcf32edd10fe4ff9a26b5140be078fda42: Status 404 returned error can't find the container with id 73f2b4181f8514c1d61ef7f1520cfabcf32edd10fe4ff9a26b5140be078fda42
Oct 31 19:30:42.143604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:42.143566    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw" event={"ID":"0f9ed52f-9c85-4647-a080-a8392a1c982b","Type":"ContainerStarted","Data":"73f2b4181f8514c1d61ef7f1520cfabcf32edd10fe4ff9a26b5140be078fda42"}
Oct 31 19:30:43.150831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:43.150800    5958 generic.go:356] "Generic (PLEG): container finished" podID="0f9ed52f-9c85-4647-a080-a8392a1c982b" containerID="8872822e1faaed006f6e385f03ec60e7635cebce8dc28028cfc94682085324f2" exitCode=0
Oct 31 19:30:43.151224 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:43.150884    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw" event={"ID":"0f9ed52f-9c85-4647-a080-a8392a1c982b","Type":"ContainerDied","Data":"8872822e1faaed006f6e385f03ec60e7635cebce8dc28028cfc94682085324f2"}
Oct 31 19:30:44.339776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.339739    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:44.437960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.437874    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k\" (UniqueName: \"kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k\") pod \"0f9ed52f-9c85-4647-a080-a8392a1c982b\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") "
Oct 31 19:30:44.438099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.438045    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca\") pod \"0f9ed52f-9c85-4647-a080-a8392a1c982b\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") "
Oct 31 19:30:44.438135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.438098    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca\") pod \"0f9ed52f-9c85-4647-a080-a8392a1c982b\" (UID: \"0f9ed52f-9c85-4647-a080-a8392a1c982b\") "
Oct 31 19:30:44.438496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.438459    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca" pod "0f9ed52f-9c85-4647-a080-a8392a1c982b" (UID: "0f9ed52f-9c85-4647-a080-a8392a1c982b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:44.440261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.440231    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca" pod "0f9ed52f-9c85-4647-a080-a8392a1c982b" (UID: "0f9ed52f-9c85-4647-a080-a8392a1c982b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:30:44.440343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.440267    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k" pod "0f9ed52f-9c85-4647-a080-a8392a1c982b" (UID: "0f9ed52f-9c85-4647-a080-a8392a1c982b"). InnerVolumeSpecName "kube-api-access-ctw7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:44.539404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.539369    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctw7k\" (UniqueName: \"kubernetes.io/projected/0f9ed52f-9c85-4647-a080-a8392a1c982b-kube-api-access-ctw7k\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:44.539404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.539402    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:44.539404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:44.539412    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ed52f-9c85-4647-a080-a8392a1c982b-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:30:45.189997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:45.189966    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"
Oct 31 19:30:45.189997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:45.189984    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw" event={"ID":"0f9ed52f-9c85-4647-a080-a8392a1c982b","Type":"ContainerDied","Data":"73f2b4181f8514c1d61ef7f1520cfabcf32edd10fe4ff9a26b5140be078fda42"}
Oct 31 19:30:45.190216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:30:45.190037    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f2b4181f8514c1d61ef7f1520cfabcf32edd10fe4ff9a26b5140be078fda42"
Oct 31 19:30:47.048093 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:30:47.048015    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:31:00.121811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:00.121772    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:31:02.673825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.673793    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"]
Oct 31 19:31:02.674569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674531    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0f9ed52f-9c85-4647-a080-a8392a1c982b" containerName="revoke-keys"
Oct 31 19:31:02.674569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674569    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9ed52f-9c85-4647-a080-a8392a1c982b" containerName="revoke-keys"
Oct 31 19:31:02.674741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674606    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:31:02.674741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674612    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:31:02.674741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674629    5958 cpu_manager.go:414] "containerMap: removing container" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:31:02.674741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674637    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:31:02.674741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674665    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:31:02.674741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674670    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:31:02.674954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674785    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35c9173-da5b-444e-ba39-18560aa5f2be" containerName="cleanup"
Oct 31 19:31:02.674954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674795    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9ed52f-9c85-4647-a080-a8392a1c982b" containerName="revoke-keys"
Oct 31 19:31:02.674954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674804    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78d82c8-c8b6-4719-b327-6e4507e9d6de" containerName="cleanup"
Oct 31 19:31:02.674954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674810    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="37407655-f69c-46c3-8c09-2d24684cf26e" containerName="cleanup"
Oct 31 19:31:02.674954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.674821    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9101d611-3a32-4ea0-b530-c58ea998821f" containerName="cleanup"
Oct 31 19:31:02.679656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.679634    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.684947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.684921    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xxh\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.684974    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685056    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685108    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685175    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685222    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685325    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685367    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.685428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.685393    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.689462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.689437    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5bbf75-openshift-default-dockercfg-qv9k6\""
Oct 31 19:31:02.692992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.692964    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"]
Oct 31 19:31:02.786165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.786128    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.786165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.786176    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.786165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.786223    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.786901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.786876    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.787155 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.787124    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.787832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.787794    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.787832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.786867    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.787844    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.787687    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.787913    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.788047    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.788066    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xxh\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788273 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.788228    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.788837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.788815    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.791207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.791176    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.792925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.792905    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.815037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.815006    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.815214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.815005    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xxh\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:02.995517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:02.995422    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:03.286791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:03.286759    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"]
Oct 31 19:31:03.287913 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:03.287875    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecffc5b_8793_4213_a825_7c78d1333e79.slice/crio-3e7b60ac6e303c0c24ccfea55a0da2fe3318337a8b8db5ea7d78607b4ff6cd41 WatchSource:0}: Error finding container 3e7b60ac6e303c0c24ccfea55a0da2fe3318337a8b8db5ea7d78607b4ff6cd41: Status 404 returned error can't find the container with id 3e7b60ac6e303c0c24ccfea55a0da2fe3318337a8b8db5ea7d78607b4ff6cd41
Oct 31 19:31:03.290388 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:03.290353    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:03.290480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:03.290437    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:03.290480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:03.290468    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:03.298578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:03.298533    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" event={"ID":"1ecffc5b-8793-4213-a825-7c78d1333e79","Type":"ContainerStarted","Data":"3e7b60ac6e303c0c24ccfea55a0da2fe3318337a8b8db5ea7d78607b4ff6cd41"}
Oct 31 19:31:04.323016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:04.322977    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" event={"ID":"1ecffc5b-8793-4213-a825-7c78d1333e79","Type":"ContainerStarted","Data":"c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7"}
Oct 31 19:31:04.996648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:04.996606    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:04.998100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:04.998072    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:31:04.998264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:04.998124    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 19:31:05.996221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:05.996181    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:31:05.996662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:05.996244    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 19:31:06.996783 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:06.996744    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:31:06.997200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:06.996799    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 19:31:07.996150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:07.996112    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:31:07.996372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:07.996183    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 19:31:08.996179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:08.996142    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:31:08.996588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:08.996202    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 19:31:10.100833 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:10.100793    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:10.101344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:10.101285    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:10.103086 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:10.102376    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:31:10.129805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:10.129746    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podStartSLOduration=8.129726816 podStartE2EDuration="8.129726816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:04.358902536 +0000 UTC m=+657.820277152" watchObservedRunningTime="2026-10-31 19:31:10.129726816 +0000 UTC m=+663.591101434"
Oct 31 19:31:12.053903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.053873    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"]
Oct 31 19:31:12.060044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.060011    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.079443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.079411    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"]
Oct 31 19:31:12.160350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.160310    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:31:12.160531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.160373    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:31:12.171055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.171025    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.171223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.171204    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.171320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.171305    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccm7l\" (UniqueName: \"kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.272663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.272610    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.272868 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.272728    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccm7l\" (UniqueName: \"kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.272868 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.272784    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.273419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.273391    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.275098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.275070    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.291117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.291086    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccm7l\" (UniqueName: \"kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l\") pod \"maas-api-revoke-sub-41294a12-zclff\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.376600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.376567    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:12.663171 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:12.663127    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0822c1_32d7_4098_8248_191261d34c1e.slice/crio-ce8d142424cc30aa9c58bfae3e30a6a559829ccf3a124dfd75428ed92e1db203 WatchSource:0}: Error finding container ce8d142424cc30aa9c58bfae3e30a6a559829ccf3a124dfd75428ed92e1db203: Status 404 returned error can't find the container with id ce8d142424cc30aa9c58bfae3e30a6a559829ccf3a124dfd75428ed92e1db203
Oct 31 19:31:12.665709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:12.665677    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"]
Oct 31 19:31:13.394892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:13.394848    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff" event={"ID":"ad0822c1-32d7-4098-8248-191261d34c1e","Type":"ContainerStarted","Data":"b09779af4f1fe7cba2c5482f4a0f1626f5b86bc5ae6bbad63b6103f52b048e81"}
Oct 31 19:31:13.395384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:13.394954    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff" event={"ID":"ad0822c1-32d7-4098-8248-191261d34c1e","Type":"ContainerStarted","Data":"ce8d142424cc30aa9c58bfae3e30a6a559829ccf3a124dfd75428ed92e1db203"}
Oct 31 19:31:13.422377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:13.422329    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff" podStartSLOduration=1.422314074 podStartE2EDuration="1.422314074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:13.420290892 +0000 UTC m=+666.881665507" watchObservedRunningTime="2026-10-31 19:31:13.422314074 +0000 UTC m=+666.883688690"
Oct 31 19:31:14.402424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.402388    5958 generic.go:356] "Generic (PLEG): container finished" podID="ad0822c1-32d7-4098-8248-191261d34c1e" containerID="b09779af4f1fe7cba2c5482f4a0f1626f5b86bc5ae6bbad63b6103f52b048e81" exitCode=0
Oct 31 19:31:14.402939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.402478    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff" event={"ID":"ad0822c1-32d7-4098-8248-191261d34c1e","Type":"ContainerDied","Data":"b09779af4f1fe7cba2c5482f4a0f1626f5b86bc5ae6bbad63b6103f52b048e81"}
Oct 31 19:31:14.598169 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.598133    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"]
Oct 31 19:31:14.602444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.602418    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.614245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.614216    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"]
Oct 31 19:31:14.703174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.703085    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.703174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.703146    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.703371 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.703273    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ztp\" (UniqueName: \"kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.805122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.805087    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.805290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.805155    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ztp\" (UniqueName: \"kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.805356 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.805339    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.805882 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.805858    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.807885 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.807861    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.824317 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.824288    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ztp\" (UniqueName: \"kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp\") pod \"maas-api-revoke-sub-c82f2c83-zhhgj\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:14.921528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:14.921245    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:15.206822 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.206776    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"]
Oct 31 19:31:15.208115 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:15.208064    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42555d54_d741_4ca3_8b29_97f9ffc4368c.slice/crio-066b051419bd8996bc189af98441ae698fb9fe950a27f1bfd19e17b69132edef WatchSource:0}: Error finding container 066b051419bd8996bc189af98441ae698fb9fe950a27f1bfd19e17b69132edef: Status 404 returned error can't find the container with id 066b051419bd8996bc189af98441ae698fb9fe950a27f1bfd19e17b69132edef
Oct 31 19:31:15.410262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.410229    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj" event={"ID":"42555d54-d741-4ca3-8b29-97f9ffc4368c","Type":"ContainerStarted","Data":"066b051419bd8996bc189af98441ae698fb9fe950a27f1bfd19e17b69132edef"}
Oct 31 19:31:15.540166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.540135    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:15.717781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.717748    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca\") pod \"ad0822c1-32d7-4098-8248-191261d34c1e\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") "
Oct 31 19:31:15.717988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.717801    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca\") pod \"ad0822c1-32d7-4098-8248-191261d34c1e\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") "
Oct 31 19:31:15.717988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.717932    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l\" (UniqueName: \"kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l\") pod \"ad0822c1-32d7-4098-8248-191261d34c1e\" (UID: \"ad0822c1-32d7-4098-8248-191261d34c1e\") "
Oct 31 19:31:15.718181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.718159    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca" pod "ad0822c1-32d7-4098-8248-191261d34c1e" (UID: "ad0822c1-32d7-4098-8248-191261d34c1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:15.718515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.718491    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:15.720289 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.720248    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca" pod "ad0822c1-32d7-4098-8248-191261d34c1e" (UID: "ad0822c1-32d7-4098-8248-191261d34c1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:31:15.720560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.720514    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l" pod "ad0822c1-32d7-4098-8248-191261d34c1e" (UID: "ad0822c1-32d7-4098-8248-191261d34c1e"). InnerVolumeSpecName "kube-api-access-ccm7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:15.833002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.823332    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0822c1-32d7-4098-8248-191261d34c1e-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:15.833002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:15.823372    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccm7l\" (UniqueName: \"kubernetes.io/projected/ad0822c1-32d7-4098-8248-191261d34c1e-kube-api-access-ccm7l\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:16.417892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.417859    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"
Oct 31 19:31:16.417892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.417874    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff" event={"ID":"ad0822c1-32d7-4098-8248-191261d34c1e","Type":"ContainerDied","Data":"ce8d142424cc30aa9c58bfae3e30a6a559829ccf3a124dfd75428ed92e1db203"}
Oct 31 19:31:16.418391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.417931    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8d142424cc30aa9c58bfae3e30a6a559829ccf3a124dfd75428ed92e1db203"
Oct 31 19:31:16.419801 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.419778    5958 generic.go:356] "Generic (PLEG): container finished" podID="42555d54-d741-4ca3-8b29-97f9ffc4368c" containerID="57f947eac4e203acee9ec1e92f8b4b0ac02374753225755118378c9cfa650c30" exitCode=0
Oct 31 19:31:16.419944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.419848    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj" event={"ID":"42555d54-d741-4ca3-8b29-97f9ffc4368c","Type":"ContainerDied","Data":"57f947eac4e203acee9ec1e92f8b4b0ac02374753225755118378c9cfa650c30"}
Oct 31 19:31:16.862232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.862197    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"]
Oct 31 19:31:16.862823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.862806    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ad0822c1-32d7-4098-8248-191261d34c1e" containerName="revoke-keys"
Oct 31 19:31:16.862879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.862826    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0822c1-32d7-4098-8248-191261d34c1e" containerName="revoke-keys"
Oct 31 19:31:16.862972 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.862961    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0822c1-32d7-4098-8248-191261d34c1e" containerName="revoke-keys"
Oct 31 19:31:16.866529 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.866504    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:16.882337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:16.882306    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"]
Oct 31 19:31:17.037671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.037632    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.037866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.037765    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wrt\" (UniqueName: \"kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.037866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.037796    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.060181 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:31:17.060131    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:31:17.139738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.139690    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wrt\" (UniqueName: \"kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.139953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.139745    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.140018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.139957    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.140749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.140721    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.142205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.142172    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.161653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.161592    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wrt\" (UniqueName: \"kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt\") pod \"maas-api-revoke-sub-a642b2ad-x85ck\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.180357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.180314    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:17.501037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.501005    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"]
Oct 31 19:31:17.624341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.624315    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:17.751126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.751048    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca\") pod \"42555d54-d741-4ca3-8b29-97f9ffc4368c\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") "
Oct 31 19:31:17.751512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.751167    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca\") pod \"42555d54-d741-4ca3-8b29-97f9ffc4368c\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") "
Oct 31 19:31:17.751512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.751296    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp\" (UniqueName: \"kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp\") pod \"42555d54-d741-4ca3-8b29-97f9ffc4368c\" (UID: \"42555d54-d741-4ca3-8b29-97f9ffc4368c\") "
Oct 31 19:31:17.752171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.752140    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca" pod "42555d54-d741-4ca3-8b29-97f9ffc4368c" (UID: "42555d54-d741-4ca3-8b29-97f9ffc4368c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:17.754613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.754587    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca" pod "42555d54-d741-4ca3-8b29-97f9ffc4368c" (UID: "42555d54-d741-4ca3-8b29-97f9ffc4368c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:31:17.754918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.754902    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp" pod "42555d54-d741-4ca3-8b29-97f9ffc4368c" (UID: "42555d54-d741-4ca3-8b29-97f9ffc4368c"). InnerVolumeSpecName "kube-api-access-f7ztp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:17.853110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.853071    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:17.853110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.853112    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42555d54-d741-4ca3-8b29-97f9ffc4368c-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:17.853110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:17.853129    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ztp\" (UniqueName: \"kubernetes.io/projected/42555d54-d741-4ca3-8b29-97f9ffc4368c-kube-api-access-f7ztp\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:18.454377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:18.454279    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck" event={"ID":"c7f656db-5838-4262-9a2b-6c5519738fcf","Type":"ContainerStarted","Data":"fcae6c84635f7926d9e5cf320382e052cceb42ebfeecd862069cb32aca3c63cc"}
Oct 31 19:31:18.454377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:18.454347    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck" event={"ID":"c7f656db-5838-4262-9a2b-6c5519738fcf","Type":"ContainerStarted","Data":"e6b43861363a2b9cdf35e045538f781ed00ab16008bd1d850db93ff8abf8c217"}
Oct 31 19:31:18.455796 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:18.455776    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"
Oct 31 19:31:18.455796 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:18.455792    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj" event={"ID":"42555d54-d741-4ca3-8b29-97f9ffc4368c","Type":"ContainerDied","Data":"066b051419bd8996bc189af98441ae698fb9fe950a27f1bfd19e17b69132edef"}
Oct 31 19:31:18.455928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:18.455832    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066b051419bd8996bc189af98441ae698fb9fe950a27f1bfd19e17b69132edef"
Oct 31 19:31:18.478204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:18.478116    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck" podStartSLOduration=2.47804959 podStartE2EDuration="2.47804959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:18.47623666 +0000 UTC m=+671.937611275" watchObservedRunningTime="2026-10-31 19:31:18.47804959 +0000 UTC m=+671.939424207"
Oct 31 19:31:19.464112 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:19.464064    5958 generic.go:356] "Generic (PLEG): container finished" podID="c7f656db-5838-4262-9a2b-6c5519738fcf" containerID="fcae6c84635f7926d9e5cf320382e052cceb42ebfeecd862069cb32aca3c63cc" exitCode=0
Oct 31 19:31:19.464537 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:19.464122    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck" event={"ID":"c7f656db-5838-4262-9a2b-6c5519738fcf","Type":"ContainerDied","Data":"fcae6c84635f7926d9e5cf320382e052cceb42ebfeecd862069cb32aca3c63cc"}
Oct 31 19:31:20.645873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.645843    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:20.701835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.701797    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca\") pod \"c7f656db-5838-4262-9a2b-6c5519738fcf\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") "
Oct 31 19:31:20.702012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.701901    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca\") pod \"c7f656db-5838-4262-9a2b-6c5519738fcf\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") "
Oct 31 19:31:20.702012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.701996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt\" (UniqueName: \"kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt\") pod \"c7f656db-5838-4262-9a2b-6c5519738fcf\" (UID: \"c7f656db-5838-4262-9a2b-6c5519738fcf\") "
Oct 31 19:31:20.702308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.702275    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca" pod "c7f656db-5838-4262-9a2b-6c5519738fcf" (UID: "c7f656db-5838-4262-9a2b-6c5519738fcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:20.702596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.702537    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:20.704665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.704639    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca" pod "c7f656db-5838-4262-9a2b-6c5519738fcf" (UID: "c7f656db-5838-4262-9a2b-6c5519738fcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:31:20.704785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.704672    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt" pod "c7f656db-5838-4262-9a2b-6c5519738fcf" (UID: "c7f656db-5838-4262-9a2b-6c5519738fcf"). InnerVolumeSpecName "kube-api-access-k2wrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:20.803898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.803812    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f656db-5838-4262-9a2b-6c5519738fcf-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:20.803898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:20.803844    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wrt\" (UniqueName: \"kubernetes.io/projected/c7f656db-5838-4262-9a2b-6c5519738fcf-kube-api-access-k2wrt\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:21.477625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:21.477592    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck" event={"ID":"c7f656db-5838-4262-9a2b-6c5519738fcf","Type":"ContainerDied","Data":"e6b43861363a2b9cdf35e045538f781ed00ab16008bd1d850db93ff8abf8c217"}
Oct 31 19:31:21.477625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:21.477624    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"
Oct 31 19:31:21.477875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:21.477665    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b43861363a2b9cdf35e045538f781ed00ab16008bd1d850db93ff8abf8c217"
Oct 31 19:31:23.280929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.280893    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"]
Oct 31 19:31:23.281530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.281514    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c7f656db-5838-4262-9a2b-6c5519738fcf" containerName="revoke-keys"
Oct 31 19:31:23.281606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.281532    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f656db-5838-4262-9a2b-6c5519738fcf" containerName="revoke-keys"
Oct 31 19:31:23.281606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.281563    5958 cpu_manager.go:414] "containerMap: removing container" podUID="42555d54-d741-4ca3-8b29-97f9ffc4368c" containerName="revoke-keys"
Oct 31 19:31:23.281606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.281569    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42555d54-d741-4ca3-8b29-97f9ffc4368c" containerName="revoke-keys"
Oct 31 19:31:23.281709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.281679    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f656db-5838-4262-9a2b-6c5519738fcf" containerName="revoke-keys"
Oct 31 19:31:23.281709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.281690    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="42555d54-d741-4ca3-8b29-97f9ffc4368c" containerName="revoke-keys"
Oct 31 19:31:23.285433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.285415    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.298518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.298490    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"]
Oct 31 19:31:23.298855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.298826    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"distinct-w6-8730d5-kserve-self-signed-certs\""
Oct 31 19:31:23.367797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.367762    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.367960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.367812    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptb8m\" (UniqueName: \"kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.367960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.367839    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.368045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.367978    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.368045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.368004    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.368197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.368176    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.469912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.469873    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.470095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.469920    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.470095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.470047    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.470236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.470141    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.470236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.470201    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptb8m\" (UniqueName: \"kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.470333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.470242    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.471128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.471097    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.471278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.471157    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.471710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.471688    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.478487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.478455    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.479249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.478846    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.494359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.494289    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptb8m\" (UniqueName: \"kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m\") pod \"distinct-w6-8730d5-kserve-f695d54f4-rdgkg\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.601722 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.601639    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:23.886229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:23.886201    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"]
Oct 31 19:31:23.887354 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:23.887328    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc33907_ee49_442f_a003_21f6425b4897.slice/crio-e2b41053a1d386cab89e7a384ee400b10adbfd7dccc89acdf577fae421988fdc WatchSource:0}: Error finding container e2b41053a1d386cab89e7a384ee400b10adbfd7dccc89acdf577fae421988fdc: Status 404 returned error can't find the container with id e2b41053a1d386cab89e7a384ee400b10adbfd7dccc89acdf577fae421988fdc
Oct 31 19:31:24.504534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:24.504493    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" event={"ID":"1bc33907-ee49-442f-a003-21f6425b4897","Type":"ContainerStarted","Data":"6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e"}
Oct 31 19:31:24.505054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:24.504581    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" event={"ID":"1bc33907-ee49-442f-a003-21f6425b4897","Type":"ContainerStarted","Data":"e2b41053a1d386cab89e7a384ee400b10adbfd7dccc89acdf577fae421988fdc"}
Oct 31 19:31:24.601964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:24.601929    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:26.102525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.102466    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podStartSLOduration=3.102452245 podStartE2EDuration="3.102452245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:24.550945394 +0000 UTC m=+678.012320009" watchObservedRunningTime="2026-10-31 19:31:26.102452245 +0000 UTC m=+679.563826860"
Oct 31 19:31:26.103488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.103469    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"]
Oct 31 19:31:26.108021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.108001    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.123102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.123073    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:31:26.123527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.123500    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:31:26.126270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.126228    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"]
Oct 31 19:31:26.211016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.210959    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdrb\" (UniqueName: \"kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.211233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.211122    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.211392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.211360    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.312752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.312712    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdrb\" (UniqueName: \"kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.312973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.312791    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.312973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.312948    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.313554 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.313519    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.315479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.315452    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.334805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.334768    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdrb\" (UniqueName: \"kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb\") pod \"maas-api-revoke-sub-fbc48236-sgvph\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.440376 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.440336    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:26.763750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:26.763718    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"]
Oct 31 19:31:26.765084 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:26.765063    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f044f0_71d8_406f_9ac0_4f9fa6ccb32c.slice/crio-195647c3d618d2b7240c5ddeb0648e981b8e51b94739ee6917604a4c522f696b WatchSource:0}: Error finding container 195647c3d618d2b7240c5ddeb0648e981b8e51b94739ee6917604a4c522f696b: Status 404 returned error can't find the container with id 195647c3d618d2b7240c5ddeb0648e981b8e51b94739ee6917604a4c522f696b
Oct 31 19:31:27.548515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:27.548476    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph" event={"ID":"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c","Type":"ContainerStarted","Data":"720216432c80e4c06c406dc7bff50e9d664423d3cebea34d6f5a050b4a6a880e"}
Oct 31 19:31:27.548946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:27.548560    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph" event={"ID":"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c","Type":"ContainerStarted","Data":"195647c3d618d2b7240c5ddeb0648e981b8e51b94739ee6917604a4c522f696b"}
Oct 31 19:31:27.573149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:27.573100    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph" podStartSLOduration=1.573085093 podStartE2EDuration="1.573085093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:27.571445185 +0000 UTC m=+681.032819805" watchObservedRunningTime="2026-10-31 19:31:27.573085093 +0000 UTC m=+681.034459750"
Oct 31 19:31:28.557436 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:28.557400    5958 generic.go:356] "Generic (PLEG): container finished" podID="68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" containerID="720216432c80e4c06c406dc7bff50e9d664423d3cebea34d6f5a050b4a6a880e" exitCode=0
Oct 31 19:31:28.557872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:28.557443    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph" event={"ID":"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c","Type":"ContainerDied","Data":"720216432c80e4c06c406dc7bff50e9d664423d3cebea34d6f5a050b4a6a880e"}
Oct 31 19:31:29.731683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.731656    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:29.761791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.761759    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca\") pod \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") "
Oct 31 19:31:29.761948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.761829    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb\" (UniqueName: \"kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb\") pod \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") "
Oct 31 19:31:29.761948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.761897    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca\") pod \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\" (UID: \"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c\") "
Oct 31 19:31:29.762274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.762249    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca" pod "68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" (UID: "68f044f0-71d8-406f-9ac0-4f9fa6ccb32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:29.762482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.762465    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:29.763936 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.763908    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb" pod "68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" (UID: "68f044f0-71d8-406f-9ac0-4f9fa6ccb32c"). InnerVolumeSpecName "kube-api-access-frdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:29.763936 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.763929    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca" pod "68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" (UID: "68f044f0-71d8-406f-9ac0-4f9fa6ccb32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:31:29.863346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.863268    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:29.863346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.863299    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdrb\" (UniqueName: \"kubernetes.io/projected/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c-kube-api-access-frdrb\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:29.929422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:29.929384    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:31:30.573790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:30.573763    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"
Oct 31 19:31:30.573964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:30.573757    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph" event={"ID":"68f044f0-71d8-406f-9ac0-4f9fa6ccb32c","Type":"ContainerDied","Data":"195647c3d618d2b7240c5ddeb0648e981b8e51b94739ee6917604a4c522f696b"}
Oct 31 19:31:30.573964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:30.573909    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195647c3d618d2b7240c5ddeb0648e981b8e51b94739ee6917604a4c522f696b"
Oct 31 19:31:32.482537 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.482502    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"]
Oct 31 19:31:32.483146 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.483131    5958 cpu_manager.go:414] "containerMap: removing container" podUID="68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" containerName="revoke-keys"
Oct 31 19:31:32.483194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.483147    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" containerName="revoke-keys"
Oct 31 19:31:32.483287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.483277    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" containerName="revoke-keys"
Oct 31 19:31:32.487345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.487322    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.504015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.503985    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:31:32.504193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.504048    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:31:32.506475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.506451    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"]
Oct 31 19:31:32.508515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.508494    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vw8j\" (UniqueName: \"kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.508651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.508555    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.508712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.508665    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.609569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.609518    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.609762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.609661    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vw8j\" (UniqueName: \"kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.609762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.609699    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.610169 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.610148    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.612097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.612067    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.624566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.624530    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vw8j\" (UniqueName: \"kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j\") pod \"maas-api-revoke-sub-7f9441f3-lsxqc\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:32.801938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:32.801860    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:33.029200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.029169    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"]
Oct 31 19:31:33.034745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.034724    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.050208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.050177    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aeb292-openshift-default-dockercfg-zq554\""
Oct 31 19:31:33.053081 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.053021    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"]
Oct 31 19:31:33.121414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121376    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121426    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121483    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121538    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121593    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121691    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6gj\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121775    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121814    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.121863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.121846    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.145735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.145681    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"]
Oct 31 19:31:33.169935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.160969    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"]
Oct 31 19:31:33.169935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.161076    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.169935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.166949    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"]
Oct 31 19:31:33.177634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.177609    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-996c32-openshift-default-dockercfg-lsfd2\""
Oct 31 19:31:33.222940 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.222910    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.222953    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.222974    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.222999    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223078    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223131    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223154    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223198    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223235    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5p7\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223260    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223327    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6gj\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223485    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223529    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223606    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223625    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223634    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.223871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223725    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223739    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.223871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223767    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.224038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223875    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.224038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.223898    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.224119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.224032    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.224119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.224086    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.225406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.225386    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.225705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.225681    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.245718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.245688    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.245891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.245873    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6gj\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.320716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.320637    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"]
Oct 31 19:31:33.325204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325171    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.325372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5p7\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.325372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325240    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.325372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325316    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.325372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325350    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325392    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325438    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325505    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325529    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325631    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325747    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325891    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325892    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.325983    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.326514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.326300    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.328064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.328039    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.328360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.328336    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.342988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.342955    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-23843f-openshift-default-dockercfg-27ptb\""
Oct 31 19:31:33.346255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.346020    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"]
Oct 31 19:31:33.349048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.348983    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.349278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.349254    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:33.351189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.351157    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5p7\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.426962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.426924    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427033    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427112    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427163    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427347 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427205    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427347 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427333    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427422    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbqd\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427643    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.427843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.427725    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.489506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.489465    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:33.529415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529378    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529424    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529445    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529472    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529490    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529604    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529683    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbqd\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529739    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529771    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.529908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.529895    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.530218 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.530184    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.530346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.530285    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.530346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.530294    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.530435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.530374    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.532080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.532053    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.532828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.532802    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.544619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.544582    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.545747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.545723    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbqd\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.596940 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.596848    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc" event={"ID":"44bae8b8-1627-49a5-962e-83a0d12dbfe2","Type":"ContainerStarted","Data":"58828fb24eb5117526a7300f5b654b7a61ed352984c4be10b58cc62c993aba0c"}
Oct 31 19:31:33.596940 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.596911    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc" event={"ID":"44bae8b8-1627-49a5-962e-83a0d12dbfe2","Type":"ContainerStarted","Data":"f6bd8babfbfdc4a3b3151d2ccb707bc4cc9e17a176d86e931da03926c98b705d"}
Oct 31 19:31:33.602289 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.602260    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:33.619075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.619047    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:33.626025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.625976    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc" podStartSLOduration=1.6259624879999999 podStartE2EDuration="1.625962488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:33.624576613 +0000 UTC m=+687.085951223" watchObservedRunningTime="2026-10-31 19:31:33.625962488 +0000 UTC m=+687.087337103"
Oct 31 19:31:33.641464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.641436    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:33.663389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.663364    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"]
Oct 31 19:31:33.663942 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:33.663904    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7572f13d_f605_4362_96dc_0c1d5575833d.slice/crio-b1357a660de53803769c9c53e2f50438165f69870a5209904f0565882c46aead WatchSource:0}: Error finding container b1357a660de53803769c9c53e2f50438165f69870a5209904f0565882c46aead: Status 404 returned error can't find the container with id b1357a660de53803769c9c53e2f50438165f69870a5209904f0565882c46aead
Oct 31 19:31:33.667808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.667778    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.667891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.667839    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.667891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.667867    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.785336 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:33.785301    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9787d41b_dc62_49f6_83e8_51ab509f73f1.slice/crio-f028a0359171b82e251f21289ab82d57c9231f08c5dc2dd981997dc444f74c22 WatchSource:0}: Error finding container f028a0359171b82e251f21289ab82d57c9231f08c5dc2dd981997dc444f74c22: Status 404 returned error can't find the container with id f028a0359171b82e251f21289ab82d57c9231f08c5dc2dd981997dc444f74c22
Oct 31 19:31:33.785470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.785442    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"]
Oct 31 19:31:33.789095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.789057    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.789204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.789148    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.789204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.789194    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.946573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.946534    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"]
Oct 31 19:31:33.960731 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.960685    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.960878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.960782    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:33.960878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:33.960832    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:34.605457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.605419    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" event={"ID":"9787d41b-dc62-49f6-83e8-51ab509f73f1","Type":"ContainerStarted","Data":"12c20372e81bd12ab0fa000de0e3bfbffba24991d8086164ce79cc47a8009015"}
Oct 31 19:31:34.606104 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.606079    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" event={"ID":"9787d41b-dc62-49f6-83e8-51ab509f73f1","Type":"ContainerStarted","Data":"f028a0359171b82e251f21289ab82d57c9231f08c5dc2dd981997dc444f74c22"}
Oct 31 19:31:34.632815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.630630    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" event={"ID":"7572f13d-f605-4362-96dc-0c1d5575833d","Type":"ContainerStarted","Data":"b08c15fe3d28932fa541217e0f62919e1d9f3b22368fb7dd08ce167567d5baad"}
Oct 31 19:31:34.632815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.630691    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" event={"ID":"7572f13d-f605-4362-96dc-0c1d5575833d","Type":"ContainerStarted","Data":"b1357a660de53803769c9c53e2f50438165f69870a5209904f0565882c46aead"}
Oct 31 19:31:34.637668 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.637636    5958 generic.go:356] "Generic (PLEG): container finished" podID="44bae8b8-1627-49a5-962e-83a0d12dbfe2" containerID="58828fb24eb5117526a7300f5b654b7a61ed352984c4be10b58cc62c993aba0c" exitCode=0
Oct 31 19:31:34.637821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.637711    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc" event={"ID":"44bae8b8-1627-49a5-962e-83a0d12dbfe2","Type":"ContainerDied","Data":"58828fb24eb5117526a7300f5b654b7a61ed352984c4be10b58cc62c993aba0c"}
Oct 31 19:31:34.639391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.639366    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" event={"ID":"3ed136f9-1352-45bd-9f0b-789ecbd43571","Type":"ContainerStarted","Data":"65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb"}
Oct 31 19:31:34.639528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.639408    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" event={"ID":"3ed136f9-1352-45bd-9f0b-789ecbd43571","Type":"ContainerStarted","Data":"5e36af7292357e322997b9c02a325796c499b70f5add01c94acfd959c43fcb34"}
Oct 31 19:31:34.641797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.641775    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:34.643386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.643369    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:31:34.649225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.649177    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podStartSLOduration=1.649161627 podStartE2EDuration="1.649161627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:34.646345873 +0000 UTC m=+688.107720491" watchObservedRunningTime="2026-10-31 19:31:34.649161627 +0000 UTC m=+688.110536241"
Oct 31 19:31:34.678148 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.678099    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podStartSLOduration=2.678085213 podStartE2EDuration="2.678085213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:34.677829821 +0000 UTC m=+688.139204434" watchObservedRunningTime="2026-10-31 19:31:34.678085213 +0000 UTC m=+688.139459828"
Oct 31 19:31:34.732363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:34.731932    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podStartSLOduration=1.731907893 podStartE2EDuration="1.731907893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:34.703301054 +0000 UTC m=+688.164675667" watchObservedRunningTime="2026-10-31 19:31:34.731907893 +0000 UTC m=+688.193282509"
Oct 31 19:31:35.351577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.350536    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:35.357791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.357745    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:31:35.358102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.358070    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 19:31:35.497644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.497603    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:35.498611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.498523    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:31:35.498611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.498603    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 19:31:35.687816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.687776    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:31:35.688284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:35.687841    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 19:31:36.391933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.391877    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:31:36.392143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.391961    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 19:31:36.402448 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.400843    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:36.493570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.493005    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:31:36.493570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.493073    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 19:31:36.493775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.493643    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca\") pod \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") "
Oct 31 19:31:36.493878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.493859    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j\" (UniqueName: \"kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j\") pod \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") "
Oct 31 19:31:36.494049 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.494028    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca\") pod \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\" (UID: \"44bae8b8-1627-49a5-962e-83a0d12dbfe2\") "
Oct 31 19:31:36.494674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.494652    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca" pod "44bae8b8-1627-49a5-962e-83a0d12dbfe2" (UID: "44bae8b8-1627-49a5-962e-83a0d12dbfe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:36.496882 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.496854    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca" pod "44bae8b8-1627-49a5-962e-83a0d12dbfe2" (UID: "44bae8b8-1627-49a5-962e-83a0d12dbfe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:31:36.497020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.497004    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j" pod "44bae8b8-1627-49a5-962e-83a0d12dbfe2" (UID: "44bae8b8-1627-49a5-962e-83a0d12dbfe2"). InnerVolumeSpecName "kube-api-access-6vw8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:36.598641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.595811    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vw8j\" (UniqueName: \"kubernetes.io/projected/44bae8b8-1627-49a5-962e-83a0d12dbfe2-kube-api-access-6vw8j\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:36.598641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.595855    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:36.598641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.595873    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bae8b8-1627-49a5-962e-83a0d12dbfe2-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:36.647247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.647210    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:31:36.647658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.647522    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 19:31:36.740780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.740745    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"
Oct 31 19:31:36.774710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.774665    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc" event={"ID":"44bae8b8-1627-49a5-962e-83a0d12dbfe2","Type":"ContainerDied","Data":"f6bd8babfbfdc4a3b3151d2ccb707bc4cc9e17a176d86e931da03926c98b705d"}
Oct 31 19:31:36.774949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:36.774930    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6bd8babfbfdc4a3b3151d2ccb707bc4cc9e17a176d86e931da03926c98b705d"
Oct 31 19:31:37.353689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:37.353652    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:31:37.353989 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:37.353955    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 19:31:37.490170 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:37.490125    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:31:37.490362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:37.490210    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 19:31:37.651570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:37.645701    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:31:37.651570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:37.645763    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 19:31:38.350570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:38.350005    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:31:38.350570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:38.350077    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 19:31:38.490205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:38.490163    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:31:38.490385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:38.490236    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 19:31:38.645576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:38.645096    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:31:38.645576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:38.645175    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 19:31:39.166648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.166608    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"]
Oct 31 19:31:39.192888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.167855    5958 cpu_manager.go:414] "containerMap: removing container" podUID="44bae8b8-1627-49a5-962e-83a0d12dbfe2" containerName="revoke-keys"
Oct 31 19:31:39.192888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.190484    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bae8b8-1627-49a5-962e-83a0d12dbfe2" containerName="revoke-keys"
Oct 31 19:31:39.192888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.190758    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bae8b8-1627-49a5-962e-83a0d12dbfe2" containerName="revoke-keys"
Oct 31 19:31:39.196876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.196844    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.197511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.197403    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"]
Oct 31 19:31:39.210284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.209834    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:31:39.210284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.209900    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:31:39.375292 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.375239    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:31:39.380149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.380084    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 19:31:39.399244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.399196    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhlqn\" (UniqueName: \"kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.399485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.399427    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.399599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.399580    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.489938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.489844    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:31:39.490075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.489927    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 19:31:39.500646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.500608    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.500771 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.500689    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.500771 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.500737    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhlqn\" (UniqueName: \"kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.501408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.501382    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.503515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.503485    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.548580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.547518    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhlqn\" (UniqueName: \"kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn\") pod \"maas-api-revoke-sub-6e5464bb-sdp2f\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:39.651583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.642017    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:31:39.651583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.642094    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 19:31:39.694929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:39.694886    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:40.364804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.364732    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:31:40.364878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.364794    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 19:31:40.367803 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.367777    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"]
Oct 31 19:31:40.491136 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.491100    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:31:40.491513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.491159    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 19:31:40.642834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.642686    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:31:40.642834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.642754    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 19:31:40.863049 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.863004    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f" event={"ID":"c7ba162b-c822-47c4-9923-9bc5430de637","Type":"ContainerStarted","Data":"ca543181671a8ba8c49849a1e5186e840cdbbfcc32e321ef5bb593d4ab87f9d1"}
Oct 31 19:31:40.863224 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.863079    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f" event={"ID":"c7ba162b-c822-47c4-9923-9bc5430de637","Type":"ContainerStarted","Data":"6b7ee468d484ffb7a1a83cb86cf9a1e80d8c6b7acb004ed191d8951c5e14e73a"}
Oct 31 19:31:40.922391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:40.922249    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f" podStartSLOduration=1.9222264930000001 podStartE2EDuration="1.922226493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:40.919528198 +0000 UTC m=+694.380902815" watchObservedRunningTime="2026-10-31 19:31:40.922226493 +0000 UTC m=+694.383601108"
Oct 31 19:31:41.501567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.497653    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:41.501567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.500182    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:41.508601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.504338    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:31:41.627391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.627351    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:41.630032 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.629965    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:41.632846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.632197    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:31:41.731387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.731350    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:41.731930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.731866    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:41.732903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.732881    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 19:31:41.871406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.871372    5958 generic.go:356] "Generic (PLEG): container finished" podID="c7ba162b-c822-47c4-9923-9bc5430de637" containerID="ca543181671a8ba8c49849a1e5186e840cdbbfcc32e321ef5bb593d4ab87f9d1" exitCode=0
Oct 31 19:31:41.871589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:41.871415    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f" event={"ID":"c7ba162b-c822-47c4-9923-9bc5430de637","Type":"ContainerDied","Data":"ca543181671a8ba8c49849a1e5186e840cdbbfcc32e321ef5bb593d4ab87f9d1"}
Oct 31 19:31:43.045024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.044998    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:43.113864 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.113826    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca\") pod \"c7ba162b-c822-47c4-9923-9bc5430de637\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") "
Oct 31 19:31:43.114151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.114127    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca\") pod \"c7ba162b-c822-47c4-9923-9bc5430de637\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") "
Oct 31 19:31:43.114263 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.114205    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn\" (UniqueName: \"kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn\") pod \"c7ba162b-c822-47c4-9923-9bc5430de637\" (UID: \"c7ba162b-c822-47c4-9923-9bc5430de637\") "
Oct 31 19:31:43.114504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.114462    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca" pod "c7ba162b-c822-47c4-9923-9bc5430de637" (UID: "c7ba162b-c822-47c4-9923-9bc5430de637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:43.114716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.114691    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:43.116345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.116312    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn" pod "c7ba162b-c822-47c4-9923-9bc5430de637" (UID: "c7ba162b-c822-47c4-9923-9bc5430de637"). InnerVolumeSpecName "kube-api-access-lhlqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:43.116443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.116366    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca" pod "c7ba162b-c822-47c4-9923-9bc5430de637" (UID: "c7ba162b-c822-47c4-9923-9bc5430de637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:31:43.216363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.216262    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhlqn\" (UniqueName: \"kubernetes.io/projected/c7ba162b-c822-47c4-9923-9bc5430de637-kube-api-access-lhlqn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:43.216363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.216306    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ba162b-c822-47c4-9923-9bc5430de637-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:31:43.923568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.923058    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"
Oct 31 19:31:43.925983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.925874    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f" event={"ID":"c7ba162b-c822-47c4-9923-9bc5430de637","Type":"ContainerDied","Data":"6b7ee468d484ffb7a1a83cb86cf9a1e80d8c6b7acb004ed191d8951c5e14e73a"}
Oct 31 19:31:43.925983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:43.925952    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7ee468d484ffb7a1a83cb86cf9a1e80d8c6b7acb004ed191d8951c5e14e73a"
Oct 31 19:31:46.384738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:46.384696    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:31:46.385236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:46.384782    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:47.057305 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:31:47.057260    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:31:47.906945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.906912    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"]
Oct 31 19:31:47.907620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.907592    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c7ba162b-c822-47c4-9923-9bc5430de637" containerName="revoke-keys"
Oct 31 19:31:47.907620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.907615    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ba162b-c822-47c4-9923-9bc5430de637" containerName="revoke-keys"
Oct 31 19:31:47.907813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.907804    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ba162b-c822-47c4-9923-9bc5430de637" containerName="revoke-keys"
Oct 31 19:31:47.912519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.912501    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:47.922166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.922138    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8b083a-openshift-default-dockercfg-jr7vh\""
Oct 31 19:31:47.928100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:47.928070    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"]
Oct 31 19:31:48.005062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005026    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005085    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdhq\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005314 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005270    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005339    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005614    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005729    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005817 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005799    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005832    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.005854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.005851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106681 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106648    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106681 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106696    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106732    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106812    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106858    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106889    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.106947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106920    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.107201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.106993    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.107201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.107043    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdhq\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.107392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.107369    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.107983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.107957    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.108311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.108276    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.108411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.108341    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.108656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.108536    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.111471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.111446    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.111681 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.111660    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.121114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.121092    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.121319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.121292    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdhq\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.232237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.232157    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:48.463617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.463582    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"]
Oct 31 19:31:48.470340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.470314    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.480080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.480012    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8730d5-openshift-default-dockercfg-hc6b7\""
Oct 31 19:31:48.483305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.483213    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"]
Oct 31 19:31:48.511134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511099    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511310 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511149    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511351    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511399    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511458    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511481    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511504    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511595    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjhd\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.511647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.511637    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.537207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.537169    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"]
Oct 31 19:31:48.538027 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:48.537989    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fea17f_4800_4daa_b01a_a82b6dcf1201.slice/crio-f418f98a70fd1bf236e82e5285a5cf508b23f41b47e731176e75eedae4c321ca WatchSource:0}: Error finding container f418f98a70fd1bf236e82e5285a5cf508b23f41b47e731176e75eedae4c321ca: Status 404 returned error can't find the container with id f418f98a70fd1bf236e82e5285a5cf508b23f41b47e731176e75eedae4c321ca
Oct 31 19:31:48.541251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.541209    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:48.541328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.541310    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:48.541409 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.541361    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:48.613274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613248    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613294    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613456 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613408    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613472    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613513    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613580    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613607    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613635    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjhd\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613661    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613702    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.613915 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.613873    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.614149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.614116    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.614354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.614332    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.614478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.614395    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.617098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.617075    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.617357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.617339    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.627198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.627140    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.627346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.627241    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjhd\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:48.789740 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:48.789654    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:49.082897 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.082868    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"]
Oct 31 19:31:49.084014 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:31:49.083980    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95555db_70a4_4383_bb42_8dae1d5e61e0.slice/crio-53f530eadbc687f659390e57f730cfacef465bccb5d20ae55d80bd118c37818b WatchSource:0}: Error finding container 53f530eadbc687f659390e57f730cfacef465bccb5d20ae55d80bd118c37818b: Status 404 returned error can't find the container with id 53f530eadbc687f659390e57f730cfacef465bccb5d20ae55d80bd118c37818b
Oct 31 19:31:49.086703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.086673    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:49.086809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.086737    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:49.086809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.086770    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:31:49.109360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.109322    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" event={"ID":"c1fea17f-4800-4daa-b01a-a82b6dcf1201","Type":"ContainerStarted","Data":"59fbbea1bfd54cd6e6a9482c4b34126776990b6e93f947803dc1d06718bb51c8"}
Oct 31 19:31:49.109624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.109594    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" event={"ID":"c1fea17f-4800-4daa-b01a-a82b6dcf1201","Type":"ContainerStarted","Data":"f418f98a70fd1bf236e82e5285a5cf508b23f41b47e731176e75eedae4c321ca"}
Oct 31 19:31:49.111091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.111062    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" event={"ID":"b95555db-70a4-4383-bb42-8dae1d5e61e0","Type":"ContainerStarted","Data":"53f530eadbc687f659390e57f730cfacef465bccb5d20ae55d80bd118c37818b"}
Oct 31 19:31:49.137708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.137654    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podStartSLOduration=2.13763789 podStartE2EDuration="2.13763789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:49.134712775 +0000 UTC m=+702.596087405" watchObservedRunningTime="2026-10-31 19:31:49.13763789 +0000 UTC m=+702.599012579"
Oct 31 19:31:49.235749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.235714    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:49.237275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.237246    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:49.237378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:49.237306    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:50.142727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.142676    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" event={"ID":"b95555db-70a4-4383-bb42-8dae1d5e61e0","Type":"ContainerStarted","Data":"3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66"}
Oct 31 19:31:50.172244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.172185    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podStartSLOduration=2.172169499 podStartE2EDuration="2.172169499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:50.170506333 +0000 UTC m=+703.631880970" watchObservedRunningTime="2026-10-31 19:31:50.172169499 +0000 UTC m=+703.633544113"
Oct 31 19:31:50.249750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.249712    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:50.249994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.249966    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:50.789921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.789861    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:50.791504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.791479    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:50.791719 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:50.791698    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:51.233003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:51.232963    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:51.233619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:51.233588    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:51.790192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:51.790152    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:51.790366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:51.790223    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:52.242565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:52.236004    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:52.242565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:52.236073    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:52.562620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:52.562479    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:31:52.790310 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:52.790265    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:52.790488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:52.790336    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:53.234031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:53.233990    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:53.234321 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:53.234293    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:53.791112 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:53.791078    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:53.791605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:53.791578    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:54.235414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:54.234963    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:54.235414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:54.235028    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:54.792496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:54.792455    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:54.793145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:54.793110    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:55.233571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:55.233105    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:55.233571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:55.233166    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:55.790127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:55.790088    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:55.790316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:55.790147    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:56.232934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:56.232888    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:31:56.233393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:56.232954    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 19:31:56.790109 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:56.790071    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:31:56.790271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:56.790141    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 19:31:57.240068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:57.240022    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:57.242164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:57.242118    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:57.243864 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:57.243825    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:31:57.797513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:57.795835    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:57.797513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:57.797354    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:31:57.798756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:31:57.798686    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:32:03.717789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.717754    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"]
Oct 31 19:32:03.723322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.723297    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.767241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.767213    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-23814d-openshift-default-dockercfg-m6pw6\""
Oct 31 19:32:03.775237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.775204    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"]
Oct 31 19:32:03.857815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.857771    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.857907    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.857966    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.857988    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.858006    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.858101    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858196 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.858168    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.858234    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ndvr\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.858294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.858283    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.959641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959602    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.959820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959678    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ndvr\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.959820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959711    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.959820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959745    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.959820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959797    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959839    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959865    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959889    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.959947    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.960068    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960292 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.960240    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.960317    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.960344    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.960603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.960540    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.962178 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.962154    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.962622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.962605    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.974671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.974612    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:03.974781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:03.974764    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ndvr\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:04.037743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:04.037702    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:04.329476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:04.329447    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"]
Oct 31 19:32:04.329949 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:04.329923    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff9953d_feea_480c_83d2_29225b3bf415.slice/crio-39f4faec28131808450461a4e383678954ca9b753fea0532a390ecb2f6652f26 WatchSource:0}: Error finding container 39f4faec28131808450461a4e383678954ca9b753fea0532a390ecb2f6652f26: Status 404 returned error can't find the container with id 39f4faec28131808450461a4e383678954ca9b753fea0532a390ecb2f6652f26
Oct 31 19:32:04.332507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:04.332478    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:04.332579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:04.332559    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:04.332619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:04.332592    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:04.357439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:04.357413    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" event={"ID":"2ff9953d-feea-480c-83d2-29225b3bf415","Type":"ContainerStarted","Data":"39f4faec28131808450461a4e383678954ca9b753fea0532a390ecb2f6652f26"}
Oct 31 19:32:05.365908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.365867    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" event={"ID":"2ff9953d-feea-480c-83d2-29225b3bf415","Type":"ContainerStarted","Data":"c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af"}
Oct 31 19:32:05.395010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.394953    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podStartSLOduration=2.394939313 podStartE2EDuration="2.394939313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:05.394468103 +0000 UTC m=+718.855842718" watchObservedRunningTime="2026-10-31 19:32:05.394939313 +0000 UTC m=+718.856313927"
Oct 31 19:32:05.560435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.560400    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"]
Oct 31 19:32:05.566789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.566761    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.572198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.572166    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"openshift-service-ca.crt\""
Oct 31 19:32:05.572711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.572688    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"default-dockercfg-n5bxr\""
Oct 31 19:32:05.572863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.572820    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"kube-root-ca.crt\""
Oct 31 19:32:05.572961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.572946    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"distinct-w4-996c32-kserve-self-signed-certs\""
Oct 31 19:32:05.576398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.576340    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"]
Oct 31 19:32:05.581292 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.581268    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.581431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.581306    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.581431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.581414    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.581670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.581652    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.581788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.581677    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.581788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.581711    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdlj\" (UniqueName: \"kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.682560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682516    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.682773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682581    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.682773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682647    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.682894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682810    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.682894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682836    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.682894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682876    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdlj\" (UniqueName: \"kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.683046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.682933    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.683095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.683053    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.683340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.683312    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.685036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.685002    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.685288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.685270    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.757567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.757442    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdlj\" (UniqueName: \"kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj\") pod \"distinct-w4-996c32-kserve-7f848c74d5-4dfqk\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:05.883635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:05.883498    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:06.039718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.039633    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:06.039987 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.039958    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:32:06.040204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.040176    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 19:32:06.246258 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.246230    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"]
Oct 31 19:32:06.248851 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:06.248818    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a744336_2742_4cfe_8920_0119d2fffc85.slice/crio-aea97301811f46a36a360f7e8921fa96b232f7486521b186c53c5a4c4bc2e5a3 WatchSource:0}: Error finding container aea97301811f46a36a360f7e8921fa96b232f7486521b186c53c5a4c4bc2e5a3: Status 404 returned error can't find the container with id aea97301811f46a36a360f7e8921fa96b232f7486521b186c53c5a4c4bc2e5a3
Oct 31 19:32:06.377254 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.377221    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" event={"ID":"5a744336-2742-4cfe-8920-0119d2fffc85","Type":"ContainerStarted","Data":"6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be"}
Oct 31 19:32:06.377254 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.377262    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" event={"ID":"5a744336-2742-4cfe-8920-0119d2fffc85","Type":"ContainerStarted","Data":"aea97301811f46a36a360f7e8921fa96b232f7486521b186c53c5a4c4bc2e5a3"}
Oct 31 19:32:06.399963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.399915    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podStartSLOduration=1.39990109 podStartE2EDuration="1.39990109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:06.398531216 +0000 UTC m=+719.859905828" watchObservedRunningTime="2026-10-31 19:32:06.39990109 +0000 UTC m=+719.861275713"
Oct 31 19:32:06.884424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:06.884391    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:07.038243 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:07.038167    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:32:07.038243 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:07.038238    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 19:32:08.049580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:08.041995    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:32:08.049580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:08.048702    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 19:32:09.042568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:09.041991    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:32:09.042568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:09.042053    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 19:32:10.043624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:10.042853    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:32:10.043624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:10.042924    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 19:32:11.038804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:11.038761    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:32:11.038988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:11.038831    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 19:32:12.386956 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:12.386890    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:12.387845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:12.387816    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:12.389508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:12.389489    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:32:15.528815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.528764    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"]
Oct 31 19:32:15.534385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.534364    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.543456 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.543430    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"]
Oct 31 19:32:15.547966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.547940    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"distinct-2-w2-aeb292-kserve-self-signed-certs\""
Oct 31 19:32:15.550031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.550006    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"openshift-service-ca.crt\""
Oct 31 19:32:15.550031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.550020    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"kube-root-ca.crt\""
Oct 31 19:32:15.550217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.550088    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"default-dockercfg-g2p8n\""
Oct 31 19:32:15.630781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.630733    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.630781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.630797    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.631050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.630913    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.631050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.630953    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktr2\" (UniqueName: \"kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.631135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.631115    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.631174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.631148    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.735809    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.735874    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.736044    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.736106    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.736160    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.736184    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktr2\" (UniqueName: \"kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.737468    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.737758    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.748091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.747636    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.749757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.749727    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.758381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.758348    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.761889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.761858    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktr2\" (UniqueName: \"kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2\") pod \"distinct-2-w2-aeb292-kserve-d995cd58f-96k4z\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.850990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.850905    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:15.884762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.883947    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:15.902516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:15.902270    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:16.151282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:16.151252    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"]
Oct 31 19:32:16.151807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:16.151786    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919fbf1d_07fd_46e3_896b_b82e7b9985bb.slice/crio-f88b4050981fc96f0461b4780667b558d4078591a3739d7c847a0f7eea2c2f35 WatchSource:0}: Error finding container f88b4050981fc96f0461b4780667b558d4078591a3739d7c847a0f7eea2c2f35: Status 404 returned error can't find the container with id f88b4050981fc96f0461b4780667b558d4078591a3739d7c847a0f7eea2c2f35
Oct 31 19:32:16.478520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:16.478434    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" event={"ID":"919fbf1d-07fd-46e3-896b-b82e7b9985bb","Type":"ContainerStarted","Data":"5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53"}
Oct 31 19:32:16.478520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:16.478479    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" event={"ID":"919fbf1d-07fd-46e3-896b-b82e7b9985bb","Type":"ContainerStarted","Data":"f88b4050981fc96f0461b4780667b558d4078591a3739d7c847a0f7eea2c2f35"}
Oct 31 19:32:16.491077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:16.491052    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:32:16.504407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:16.504358    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podStartSLOduration=1.5043459590000001 podStartE2EDuration="1.504345959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:16.500466205 +0000 UTC m=+729.961840819" watchObservedRunningTime="2026-10-31 19:32:16.504345959 +0000 UTC m=+729.965720571"
Oct 31 19:32:16.851863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:16.851776    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:17.047529 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:32:17.047485    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:32:18.106434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.106402    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"]
Oct 31 19:32:18.111857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.111832    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.124819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.124789    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"]
Oct 31 19:32:18.189806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.189771    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.189974    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190007    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgflc\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190082    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190115    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190141    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190278    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190317    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.190506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.190401    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.302749    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.302798    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgflc\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.302850    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.302883    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.302918    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.302981    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.303004    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.303057    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.303162    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.305974    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.309630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.306239    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.319891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.317916    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.319891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.318047    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"]
Oct 31 19:32:18.320119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.320097    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.320665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.320607    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.321185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.321161    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.322818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.322796    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgflc\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.322953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.322929    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.328227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.328202    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.334590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.328095    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"]
Oct 31 19:32:18.341285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.341248    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.404444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404317    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404379    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgm8n\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404426    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404441    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404469    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404488    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404524    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404581    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.404849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.404658    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.420222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.420190    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"]
Oct 31 19:32:18.425318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.425289    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.430290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.430266    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:18.447670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.447632    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"]
Oct 31 19:32:18.506123 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506044    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.506123 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506087    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506351    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506385    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506405    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgm8n\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506645    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506697    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506719    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506759    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506788    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506818    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506860    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506905    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.506932    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx8j\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.507007    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.507052    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.507096    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.507134    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.507474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.507147    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.508962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.508511    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.508962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.508845    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.508962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.508904    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.509199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.509114    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.535017 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.518521    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.535017 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.524423    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgm8n\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.535017 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.528121    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.552507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.552465    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.608443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.608393    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.608660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.608462    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.608660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.608514    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.608814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.608680    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.608987 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.608960    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609347    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609467    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609567    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx8j\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609679    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609726    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609761    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.609832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.609821    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.610072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.610044    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.610156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.610135    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.611316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.611289    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.612313 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.612185    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.631240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.631157    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.631240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.631225    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx8j\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.677747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.677714    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:18.769584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.769519    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"]
Oct 31 19:32:18.770774 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:18.770741    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdca189_5947_4c03_a1f8_790bf8c6a68d.slice/crio-a356cadb44d63d7a05c1ebf61546fa61750d91fc09d5231b7cdb2acb6d7e53c5 WatchSource:0}: Error finding container a356cadb44d63d7a05c1ebf61546fa61750d91fc09d5231b7cdb2acb6d7e53c5: Status 404 returned error can't find the container with id a356cadb44d63d7a05c1ebf61546fa61750d91fc09d5231b7cdb2acb6d7e53c5
Oct 31 19:32:18.774629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.774594    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:18.774734 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.774685    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:18.774734 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.774723    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:18.792864 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.792826    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:18.972796 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.972757    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"]
Oct 31 19:32:18.972796 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:18.972777    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e5a527_69f8_4d09_b89b_55d576637e7d.slice/crio-0e490cbfa52e66263c88136f090e1cfba86ae5747a736e2b4754806af8ed8283 WatchSource:0}: Error finding container 0e490cbfa52e66263c88136f090e1cfba86ae5747a736e2b4754806af8ed8283: Status 404 returned error can't find the container with id 0e490cbfa52e66263c88136f090e1cfba86ae5747a736e2b4754806af8ed8283
Oct 31 19:32:18.975682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.975651    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:18.975775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.975727    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:18.975775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:18.975756    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:19.104785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.104742    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"]
Oct 31 19:32:19.115916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.114577    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:19.115916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.114666    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:19.115916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.114706    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:19.559311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.559056    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" event={"ID":"c8e5a527-69f8-4d09-b89b-55d576637e7d","Type":"ContainerStarted","Data":"af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e"}
Oct 31 19:32:19.559311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.559178    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" event={"ID":"c8e5a527-69f8-4d09-b89b-55d576637e7d","Type":"ContainerStarted","Data":"0e490cbfa52e66263c88136f090e1cfba86ae5747a736e2b4754806af8ed8283"}
Oct 31 19:32:19.561237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.561198    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" event={"ID":"1026494f-1bd1-4171-96cb-f4322fe75f65","Type":"ContainerStarted","Data":"684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9"}
Oct 31 19:32:19.561369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.561271    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" event={"ID":"1026494f-1bd1-4171-96cb-f4322fe75f65","Type":"ContainerStarted","Data":"1964d1bf4c27785b8c51b6cafda3e983735779231d3aa31f00e60a678f98a0ca"}
Oct 31 19:32:19.563033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.563006    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" event={"ID":"9fdca189-5947-4c03-a1f8-790bf8c6a68d","Type":"ContainerStarted","Data":"e2a5d2a0b4b0c02a7655a98835ee6a7ff633ca6317bf7666aad14bfd2127ca99"}
Oct 31 19:32:19.563134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.563051    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" event={"ID":"9fdca189-5947-4c03-a1f8-790bf8c6a68d","Type":"ContainerStarted","Data":"a356cadb44d63d7a05c1ebf61546fa61750d91fc09d5231b7cdb2acb6d7e53c5"}
Oct 31 19:32:19.596576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.596487    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podStartSLOduration=1.596469367 podStartE2EDuration="1.596469367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:19.588509003 +0000 UTC m=+733.049883619" watchObservedRunningTime="2026-10-31 19:32:19.596469367 +0000 UTC m=+733.057843995"
Oct 31 19:32:19.659908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.655824    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podStartSLOduration=1.655803931 podStartE2EDuration="1.655803931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:19.63096281 +0000 UTC m=+733.092337463" watchObservedRunningTime="2026-10-31 19:32:19.655803931 +0000 UTC m=+733.117178547"
Oct 31 19:32:19.678403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.678364    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:19.680015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.679959    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podStartSLOduration=1.67994236 podStartE2EDuration="1.67994236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:19.676521035 +0000 UTC m=+733.137895652" watchObservedRunningTime="2026-10-31 19:32:19.67994236 +0000 UTC m=+733.141316974"
Oct 31 19:32:19.792990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:19.792904    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:20.434356 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.434319    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:20.435099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.435073    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:32:20.435283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.435260    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:32:20.695045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.694939    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 19:32:20.695045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.695002    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 19:32:20.794528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.794495    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 19:32:20.794728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:20.794564    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 19:32:21.437570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:21.433658    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:32:21.437570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:21.433724    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:32:21.678667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:21.678194    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 19:32:21.678667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:21.678260    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 19:32:21.794024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:21.793931    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 19:32:21.794287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:21.794257    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 19:32:22.441569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:22.440574    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:32:22.441569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:22.440649    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:32:22.688790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:22.688691    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 19:32:22.688790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:22.688761    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 19:32:22.793707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:22.793632    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 19:32:22.793924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:22.793895    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 19:32:23.488977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.488936    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:32:23.489481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.489000    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:32:23.693571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.693073    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 19:32:23.693571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.693171    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 19:32:23.797015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.796930    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 19:32:23.797304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.797264    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 19:32:23.930567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.903908    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"]
Oct 31 19:32:23.950576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:23.935470    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.034607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.015487    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"]
Oct 31 19:32:24.104896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.083333    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.104896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.083453    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs8v\" (UniqueName: \"kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.104896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.083556    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.104896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.083597    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.104896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.083679    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.104896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.083702    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.134583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.121403    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"openshift-service-ca.crt\""
Oct 31 19:32:24.135177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.135146    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"distinct-2-w5-5bbf75-kserve-self-signed-certs\""
Oct 31 19:32:24.135458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.135443    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"kube-root-ca.crt\""
Oct 31 19:32:24.135655 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.135642    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"default-dockercfg-8kc45\""
Oct 31 19:32:24.204578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.187725    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.204990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.204957    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs8v\" (UniqueName: \"kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.205215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.205196    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.262364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.260941    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.291173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.261115    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.291410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.273481    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.294625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.291131    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.320657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.320627    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.325706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.325674    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.329566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.326085    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.344729 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.333424    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs8v\" (UniqueName: \"kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.369627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.365238    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs\") pod \"distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.426095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.425796    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:24.431732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.431692    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:32:24.446352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.446303    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:32:24.722798 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.722696    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 19:32:24.723379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.723346    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 19:32:24.877578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.876748    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 19:32:24.877578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:24.876819    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 19:32:25.226569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.203873    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"]
Oct 31 19:32:25.227389 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:25.227328    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9627c756_873f_4326_90a0_190da4caf84a.slice/crio-9a6aa5d1012d0fcf80c3da75b3eff326ca16ab03398c666c451c91e7ab201633 WatchSource:0}: Error finding container 9a6aa5d1012d0fcf80c3da75b3eff326ca16ab03398c666c451c91e7ab201633: Status 404 returned error can't find the container with id 9a6aa5d1012d0fcf80c3da75b3eff326ca16ab03398c666c451c91e7ab201633
Oct 31 19:32:25.446608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.437812    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:32:25.446608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.437879    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:32:25.678348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.678305    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 19:32:25.678523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.678392    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 19:32:25.793358 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.793265    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 19:32:25.793358 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.793333    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 19:32:25.851271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.851229    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:25.871203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.871163    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:25.924723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.924182    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" event={"ID":"9627c756-873f-4326-90a0-190da4caf84a","Type":"ContainerStarted","Data":"dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e"}
Oct 31 19:32:25.924723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.924231    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" event={"ID":"9627c756-873f-4326-90a0-190da4caf84a","Type":"ContainerStarted","Data":"9a6aa5d1012d0fcf80c3da75b3eff326ca16ab03398c666c451c91e7ab201633"}
Oct 31 19:32:25.982324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.981334    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podStartSLOduration=2.9813108870000002 podStartE2EDuration="2.981310887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:25.951349772 +0000 UTC m=+739.412724390" watchObservedRunningTime="2026-10-31 19:32:25.981310887 +0000 UTC m=+739.442685504"
Oct 31 19:32:25.994842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:25.994811    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:32:26.427019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.426974    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:26.775651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.769711    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:26.775651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.770977    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:26.775651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.774499    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 19:32:26.878952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.878915    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:26.880736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.880710    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:26.884434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.884405    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:32:26.917312 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.917276    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:26.917922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.917898    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:26.919592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:26.919568    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:32:29.933604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:29.933563    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:32:32.894575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.893607    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"]
Oct 31 19:32:32.907670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.907592    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.961605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.930047    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.994996    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995066    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995127    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995160    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvj9\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995243    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995281    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995606    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:32.995935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:32.995658    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.002569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.001435    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"]
Oct 31 19:32:33.097786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.097752    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.098181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.098163    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.098347 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.098331    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.098518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.098504    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.098646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.098628    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvj9\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.098792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.098778    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.098889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.098877    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.099109 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.099096    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.099203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.099191    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.099848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.099825    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.100195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.100179    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.104357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.104329    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.106130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.106102    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.107293 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.107271    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.111352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.111327    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.195935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.195443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.214596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.205324    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.244606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.244432    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvj9\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.430567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.430074    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:33.701149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.697185    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"]
Oct 31 19:32:33.757092 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.756455    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.810637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.810604    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"]
Oct 31 19:32:33.816506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.816447    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:33.816969 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.816826    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:33.817227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.817190    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:33.823361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.823336    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"]
Oct 31 19:32:33.843419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.843389    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.848877 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.847372    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"]
Oct 31 19:32:33.864306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.864275    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"]
Oct 31 19:32:33.877444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877418    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877451    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877601    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877712    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9xsb\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877755    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877775    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877789    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.877858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.877826    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.979577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979504    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.979577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979574    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979622    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979732    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979773    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979802    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979885    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6vx\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979923    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979931    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.979948    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980028    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980057    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980443    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980481    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980525    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9xsb\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980608    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980638    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980640    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.980697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980695    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.981046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980749    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:33.981046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.980954    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.981151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.981104    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.981215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.981193    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.983514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.983479    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.985026 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.985005    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.994448 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.994418    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:33.994760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:33.994735    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9xsb\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:34.082434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082337    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082405    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082439    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082465    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082537    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6vx\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082631    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082770    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082799    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082853    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.082910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082859    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.083173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.082914    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.083173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.083082    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.083281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.083248    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.083621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.083592    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.086003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.085977    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.086299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.086275    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.096834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.096809    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.097487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.097459    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6vx\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.107581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.107527    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" event={"ID":"5b10670d-0976-4783-be38-d7dc50ee95b4","Type":"ContainerStarted","Data":"a9f5cefb70143207fae5142fb03152ebba0228395ae244f97f6535486e13b2ed"}
Oct 31 19:32:34.107736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.107614    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" event={"ID":"5b10670d-0976-4783-be38-d7dc50ee95b4","Type":"ContainerStarted","Data":"6eb95a2bf7ee7a2a5bd3c65efd6341b253df778701f959342ba8e2c2c490e31f"}
Oct 31 19:32:34.124895 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.124766    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:34.135156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.135082    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podStartSLOduration=2.13506339 podStartE2EDuration="2.13506339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:34.133696368 +0000 UTC m=+747.595070987" watchObservedRunningTime="2026-10-31 19:32:34.13506339 +0000 UTC m=+747.596438008"
Oct 31 19:32:34.182124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.182089    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:34.433436 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.433272    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"]
Oct 31 19:32:34.433614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.433514    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:34.434214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.434185    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:34.435630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.435604    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:34.435736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.435656    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:34.437470 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:34.437447    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdec5e7_c13b_4bf1_9e24_b4d2585f2466.slice/crio-f0791e9cc87f6b7ea829d3b69db151ccd2bd5c34eed23a7b9310a6f02beeac82 WatchSource:0}: Error finding container f0791e9cc87f6b7ea829d3b69db151ccd2bd5c34eed23a7b9310a6f02beeac82: Status 404 returned error can't find the container with id f0791e9cc87f6b7ea829d3b69db151ccd2bd5c34eed23a7b9310a6f02beeac82
Oct 31 19:32:34.440414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.440381    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:34.440511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.440461    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:34.440608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.440508    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:34.448216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.448191    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:34.535814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.535757    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"]
Oct 31 19:32:34.541443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.541391    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:34.541645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.541485    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:34.541645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:34.541582    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:35.151651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.150904    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" event={"ID":"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a","Type":"ContainerStarted","Data":"e6e17427989aec2597d54342b905063f79d85608275c1986d78a46e0852d0c03"}
Oct 31 19:32:35.151651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.150958    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" event={"ID":"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a","Type":"ContainerStarted","Data":"a9b63f53783f99b85b71926a60fcb5a3371746b9c39ebe3a4a5e96b49357d152"}
Oct 31 19:32:35.161893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.155672    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" event={"ID":"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466","Type":"ContainerStarted","Data":"0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1"}
Oct 31 19:32:35.161893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.155729    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" event={"ID":"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466","Type":"ContainerStarted","Data":"f0791e9cc87f6b7ea829d3b69db151ccd2bd5c34eed23a7b9310a6f02beeac82"}
Oct 31 19:32:35.194237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.194197    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:35.194653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.194627    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:32:35.199271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.199200    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podStartSLOduration=2.199178432 podStartE2EDuration="2.199178432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:35.19084262 +0000 UTC m=+748.652217261" watchObservedRunningTime="2026-10-31 19:32:35.199178432 +0000 UTC m=+748.660553048"
Oct 31 19:32:35.200072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.200041    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:35.200323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.200293    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:35.441685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.431023    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:35.441685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.431088    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:35.494565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:35.457197    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podStartSLOduration=2.457172428 podStartE2EDuration="2.457172428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:35.256063799 +0000 UTC m=+748.717438443" watchObservedRunningTime="2026-10-31 19:32:35.457172428 +0000 UTC m=+748.918547045"
Oct 31 19:32:36.130567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.130515    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:36.130989 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.130964    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:36.131152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.131129    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:36.192584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.188575    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:36.192584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.188806    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:36.446626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.440605    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:36.446626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:36.440667    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:37.151134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:37.151095    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:37.152277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:37.152233    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:37.205570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:37.205393    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:37.205570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:37.205506    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:37.439216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:37.433024    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:37.439216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:37.433084    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:38.127571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:38.126919    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:38.127571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:38.126979    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:38.184568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:38.183955    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:38.184568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:38.184023    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:38.432625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:38.432085    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:38.432625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:38.432154    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:39.133568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:39.132963    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:39.133568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:39.133036    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:39.183390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:39.182906    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:39.183390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:39.182970    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:39.432052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:39.431950    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:39.432381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:39.432348    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:40.126569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:40.126047    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:40.126569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:40.126107    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:40.182577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:40.182489    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:40.182783 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:40.182577    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:40.453624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:40.438671    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:32:40.453624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:40.438758    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 31 19:32:41.129577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.127724    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:41.129577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.127797    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:41.183463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.183432    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:41.183725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.183703    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:41.766464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.766420    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:41.768012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.767978    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:41.769794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:41.769766    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:32:42.145995 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:42.145952    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:32:42.146736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:42.146687    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 19:32:42.192366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:42.192320    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 31 19:32:42.192682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:42.192648    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 31 19:32:43.132132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:43.132096    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:43.133163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:43.132725    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:43.134127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:43.134104    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:32:43.187600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:43.187534    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:43.187990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:43.187845    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:43.188648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:43.188630    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:32:45.627572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.627527    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"]
Oct 31 19:32:45.632996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.632971    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.638527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.638496    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"]
Oct 31 19:32:45.644621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.644538    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"default-dockercfg-55sdv\""
Oct 31 19:32:45.644760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.644650    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"kube-root-ca.crt\""
Oct 31 19:32:45.644760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.644652    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"distinct-2-w1-8b083a-kserve-self-signed-certs\""
Oct 31 19:32:45.644760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.644707    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"openshift-service-ca.crt\""
Oct 31 19:32:45.740754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.740710    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.740938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.740884    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.740938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.740914    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjp5\" (UniqueName: \"kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.741207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.741186    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.741256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.741227    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.741396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.741358    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.851568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.849489    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.851568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.849558    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjp5\" (UniqueName: \"kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.851568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.849818    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.851568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.849870    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.851568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.849961    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.851568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.850018    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.852034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.851852    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.852197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.852173    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.854903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.854869    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.884281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.884200    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.885997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.885036    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjp5\" (UniqueName: \"kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.905569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.902414    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm\") pod \"distinct-2-w1-8b083a-kserve-67d646f857-d4wgn\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:45.955849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:45.955814    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:46.245273 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:46.245237    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"]
Oct 31 19:32:46.245679 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:46.245652    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a5c4855_d499_4812_99ec_da312af9e686.slice/crio-fed0e2976e15438544f6211231b65a6fa101cbb855160820e84f8c8d74d76a54 WatchSource:0}: Error finding container fed0e2976e15438544f6211231b65a6fa101cbb855160820e84f8c8d74d76a54: Status 404 returned error can't find the container with id fed0e2976e15438544f6211231b65a6fa101cbb855160820e84f8c8d74d76a54
Oct 31 19:32:46.528191 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:46.528100    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" event={"ID":"4a5c4855-d499-4812-99ec-da312af9e686","Type":"ContainerStarted","Data":"bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea"}
Oct 31 19:32:46.528191 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:46.528151    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" event={"ID":"4a5c4855-d499-4812-99ec-da312af9e686","Type":"ContainerStarted","Data":"fed0e2976e15438544f6211231b65a6fa101cbb855160820e84f8c8d74d76a54"}
Oct 31 19:32:46.552087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:46.552029    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podStartSLOduration=1.552014459 podStartE2EDuration="1.552014459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:46.550346878 +0000 UTC m=+760.011721497" watchObservedRunningTime="2026-10-31 19:32:46.552014459 +0000 UTC m=+760.013389071"
Oct 31 19:32:46.957069 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:46.957037    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:47.045300 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:32:47.045262    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:32:48.219994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.219949    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"]
Oct 31 19:32:48.264741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.264706    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"]
Oct 31 19:32:48.265179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.265097    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.330617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330570    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.330810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330635    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.330810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330760    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.330942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330812    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.330942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330878    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.330942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330916    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.331093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330964    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.331093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.330995    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.331093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.331075    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432175    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432245    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432306    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432341    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432387    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432413    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432481    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432635    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.432812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.432672    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.433101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.433076    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.433101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.433095    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.433362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.433341    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.433863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.433783    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.433990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.433915    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.436373 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.436341    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.438500 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.437765    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.447739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.447708    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.448385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.448359    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.608614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.602841    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:48.813392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.813351    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"]
Oct 31 19:32:48.907570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.885736    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"]
Oct 31 19:32:48.907570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:48.885863    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011658    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24qr\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011730    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011786    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011838    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011891    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011927    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.011980    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.012023    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.032089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.012203    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.149493    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.149573    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.149637    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.149685    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.149766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.149956    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24qr\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.150001    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.150057    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.150106    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.150754    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.151016    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.187101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.172958    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.224338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.224035    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.240577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.239397    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.368208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.354327    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.368208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.361508    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24qr\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.390611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.379011    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.390611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.390370    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.448705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.434146    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:49.620430 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.620395    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"]
Oct 31 19:32:49.692821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.690726    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:49.692821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.690999    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:49.692821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.691161    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:49.977997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.977961    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"]
Oct 31 19:32:49.980089 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:32:49.979645    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34092465_27ca_4013_a8e8_1134623d0f4e.slice/crio-11c455bf4c59e234c48d329afdcdab75cce6ddc2e1652d9f7e3aa0ede10e9c45 WatchSource:0}: Error finding container 11c455bf4c59e234c48d329afdcdab75cce6ddc2e1652d9f7e3aa0ede10e9c45: Status 404 returned error can't find the container with id 11c455bf4c59e234c48d329afdcdab75cce6ddc2e1652d9f7e3aa0ede10e9c45
Oct 31 19:32:49.983526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.983487    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:49.983704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.983590    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:49.983704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:49.983664    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:32:50.639781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:50.639721    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" event={"ID":"34092465-27ca-4013-a8e8-1134623d0f4e","Type":"ContainerStarted","Data":"3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14"}
Oct 31 19:32:50.640373 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:50.640344    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" event={"ID":"34092465-27ca-4013-a8e8-1134623d0f4e","Type":"ContainerStarted","Data":"11c455bf4c59e234c48d329afdcdab75cce6ddc2e1652d9f7e3aa0ede10e9c45"}
Oct 31 19:32:50.641867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:50.641838    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" event={"ID":"9c9469d9-55d6-4932-89e6-23a41a08e600","Type":"ContainerStarted","Data":"b959a199a64da8d68195b01302234b5084925a1f24269b792ffa8b846de6b67e"}
Oct 31 19:32:50.642009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:50.641886    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" event={"ID":"9c9469d9-55d6-4932-89e6-23a41a08e600","Type":"ContainerStarted","Data":"f9463ea97ab90755bfdd4c4941166edb13b8f80dd8896966cd2b4221de426abe"}
Oct 31 19:32:50.674713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:50.674655    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podStartSLOduration=2.67464076 podStartE2EDuration="2.67464076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:50.67121232 +0000 UTC m=+764.132586965" watchObservedRunningTime="2026-10-31 19:32:50.67464076 +0000 UTC m=+764.136015449"
Oct 31 19:32:50.705299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:50.705213    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podStartSLOduration=2.705188235 podStartE2EDuration="2.705188235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:50.699830235 +0000 UTC m=+764.161204851" watchObservedRunningTime="2026-10-31 19:32:50.705188235 +0000 UTC m=+764.166562852"
Oct 31 19:32:51.445332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:51.445295    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:51.448355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:51.447226    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:51.448355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:51.447291    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:51.603509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:51.603471    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:51.619107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:51.615602    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:51.619107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:51.615665    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:52.452569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:52.449971    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:52.452569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:52.450035    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:52.605013 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:52.604969    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:52.605302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:52.605271    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:53.442568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:53.441935    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:53.442568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:53.441997    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:53.603863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:53.603826    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:53.604449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:53.604411    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:54.467526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:54.467482    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:54.467740 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:54.467556    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:54.612595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:54.607941    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:54.612595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:54.608007    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:55.453570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:55.453124    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:55.453570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:55.453281    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:55.629605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:55.623136    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:55.629605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:55.623210    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:55.959610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:55.959355    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:55.972162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:55.972131    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:56.444034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:56.443995    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:56.444311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:56.444283    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:56.616720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:56.616681    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:56.616918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:56.616766    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:56.732386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:56.732291    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:32:57.441360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:57.441318    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.441616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:57.441427    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 19:32:57.604198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:57.604159    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.604370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:57.604227    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 19:32:58.445835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:58.445805    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:58.446304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:58.446107    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:58.446851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:58.446832    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 19:32:58.608892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:58.608861    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:58.609296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:58.609276    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:58.610137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:58.610116    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:32:59.371173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.371136    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"]
Oct 31 19:32:59.377366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.377343    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.379612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.379539    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.379809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.379773    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggz7\" (UniqueName: \"kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.380025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.379998    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.382764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.382672    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:32:59.382764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.382734    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:32:59.384941 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.384921    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"]
Oct 31 19:32:59.482056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.482012    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.482559 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.482134    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.482559 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.482266    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggz7\" (UniqueName: \"kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.482815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.482792    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.484666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.484641    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.495951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.495921    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggz7\" (UniqueName: \"kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7\") pod \"maas-api-revoke-sub-0b246870-7vlr8\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:32:59.708620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:32:59.702913    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:33:00.039306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:00.039273    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"]
Oct 31 19:33:00.785349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:00.785256    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8" event={"ID":"a0a81b0d-b809-4fa1-8721-b6b24ceedda5","Type":"ContainerStarted","Data":"69b7f2c3687c88e68674e6104873e8c7c4386d970fd32b0e6e8be8c0858a53da"}
Oct 31 19:33:00.785349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:00.785328    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8" event={"ID":"a0a81b0d-b809-4fa1-8721-b6b24ceedda5","Type":"ContainerStarted","Data":"7e7d901b0b3e6403573ca7b75f7ff775a00e9da06f873774202e188894229eeb"}
Oct 31 19:33:00.805539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:00.805491    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8" podStartSLOduration=1.805476975 podStartE2EDuration="1.805476975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:00.804896785 +0000 UTC m=+774.266271400" watchObservedRunningTime="2026-10-31 19:33:00.805476975 +0000 UTC m=+774.266851589"
Oct 31 19:33:01.794101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:01.794067    5958 generic.go:356] "Generic (PLEG): container finished" podID="a0a81b0d-b809-4fa1-8721-b6b24ceedda5" containerID="69b7f2c3687c88e68674e6104873e8c7c4386d970fd32b0e6e8be8c0858a53da" exitCode=0
Oct 31 19:33:01.794594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:01.794121    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8" event={"ID":"a0a81b0d-b809-4fa1-8721-b6b24ceedda5","Type":"ContainerDied","Data":"69b7f2c3687c88e68674e6104873e8c7c4386d970fd32b0e6e8be8c0858a53da"}
Oct 31 19:33:03.036775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.014720    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"]
Oct 31 19:33:03.056950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.056866    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"]
Oct 31 19:33:03.057288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.057264    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.082581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.082496    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.082757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.082624    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.082867    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmxf\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.082941    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.082993    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.083066    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.083139    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.083192    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.083571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.083274    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.155625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.155598    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:33:03.184826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.184778    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca\") pod \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") "
Oct 31 19:33:03.185024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.184914    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7\" (UniqueName: \"kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7\") pod \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") "
Oct 31 19:33:03.185096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185019    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca\") pod \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\" (UID: \"a0a81b0d-b809-4fa1-8721-b6b24ceedda5\") "
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185385    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185460    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185497    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185570    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185600    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca" pod "a0a81b0d-b809-4fa1-8721-b6b24ceedda5" (UID: "a0a81b0d-b809-4fa1-8721-b6b24ceedda5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185615    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.185720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.185688    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186257    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186328    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186465    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186576    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmxf\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186631    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186650    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186680    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.186996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.186796    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:03.187459 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.187135    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.188793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.188766    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7" pod "a0a81b0d-b809-4fa1-8721-b6b24ceedda5" (UID: "a0a81b0d-b809-4fa1-8721-b6b24ceedda5"). InnerVolumeSpecName "kube-api-access-mggz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:03.190093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.190066    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.190291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.190141    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.192166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.192125    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca" pod "a0a81b0d-b809-4fa1-8721-b6b24ceedda5" (UID: "a0a81b0d-b809-4fa1-8721-b6b24ceedda5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:33:03.200885 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.200856    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmxf\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.201026 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.201002    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.287965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.287933    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mggz7\" (UniqueName: \"kubernetes.io/projected/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-kube-api-access-mggz7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:03.287965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.287963    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a81b0d-b809-4fa1-8721-b6b24ceedda5-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:03.409573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.409508    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:03.493615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.492843    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"]
Oct 31 19:33:03.494400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.493923    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a0a81b0d-b809-4fa1-8721-b6b24ceedda5" containerName="revoke-keys"
Oct 31 19:33:03.494400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.493945    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a81b0d-b809-4fa1-8721-b6b24ceedda5" containerName="revoke-keys"
Oct 31 19:33:03.494400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.494158    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a81b0d-b809-4fa1-8721-b6b24ceedda5" containerName="revoke-keys"
Oct 31 19:33:03.500656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.500627    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.536599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.530277    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"]
Oct 31 19:33:03.593713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.593671    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.593863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.593815    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.593863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.593842    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.593959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.593881    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.594080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.594043    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.594080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.594076    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.594197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.594177    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.594234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.594206    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrlq\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.594265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.594248    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696115    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696193    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696211    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696267    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696283    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrlq\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696340    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696470    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696618    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696639    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696652    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696684    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.696944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696865    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.697002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.696943    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.697354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.697329    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.698730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.698707    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.699063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.699041    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.705498 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:03.705471    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39a418e_9e97_41cb_8753_5fe2c2f8716b.slice/crio-234b85f78d475020e6a4f48bb51b2ef3c9752e62f74789cf1fdd3d28f034e8b7 WatchSource:0}: Error finding container 234b85f78d475020e6a4f48bb51b2ef3c9752e62f74789cf1fdd3d28f034e8b7: Status 404 returned error can't find the container with id 234b85f78d475020e6a4f48bb51b2ef3c9752e62f74789cf1fdd3d28f034e8b7
Oct 31 19:33:03.706346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.706323    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"]
Oct 31 19:33:03.708996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.708957    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:03.709075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.709051    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:03.709143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.709126    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:03.709954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.709937    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.710052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.710031    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrlq\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-88shr\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.846170 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.845893    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:03.935618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.923569    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8" event={"ID":"a0a81b0d-b809-4fa1-8721-b6b24ceedda5","Type":"ContainerDied","Data":"7e7d901b0b3e6403573ca7b75f7ff775a00e9da06f873774202e188894229eeb"}
Oct 31 19:33:03.935618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.923639    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e7d901b0b3e6403573ca7b75f7ff775a00e9da06f873774202e188894229eeb"
Oct 31 19:33:03.935618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.923723    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"
Oct 31 19:33:03.935618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.930400    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" event={"ID":"b39a418e-9e97-41cb-8753-5fe2c2f8716b","Type":"ContainerStarted","Data":"d95998a1a3b330c0a06c448a9735af0196546d036be6a4ef20f3c39bfbc7d7ba"}
Oct 31 19:33:03.935618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.930460    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" event={"ID":"b39a418e-9e97-41cb-8753-5fe2c2f8716b","Type":"ContainerStarted","Data":"234b85f78d475020e6a4f48bb51b2ef3c9752e62f74789cf1fdd3d28f034e8b7"}
Oct 31 19:33:03.966718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.966650    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podStartSLOduration=1.9666106810000001 podStartE2EDuration="1.966610681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:03.957885795 +0000 UTC m=+777.419260426" watchObservedRunningTime="2026-10-31 19:33:03.966610681 +0000 UTC m=+777.427985297"
Oct 31 19:33:03.998611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:03.995793    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"]
Oct 31 19:33:04.011529 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.011501    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.016182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.016135    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"]
Oct 31 19:33:04.031993 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.029329    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:33:04.031993 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.029418    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:33:04.142267 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.142201    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.142923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.142647    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.142923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.142749    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjp9\" (UniqueName: \"kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.244135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.244103    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjp9\" (UniqueName: \"kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.244336 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.244316    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.245800 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.245766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.245979 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.245949    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"]
Oct 31 19:33:04.246180 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:04.246149    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f705aa_c316_4b3f_a179_9f65434398c3.slice/crio-2ae02c18f127dd6840789dec41edc280ebdde74982eb17b1719dc41d02032028 WatchSource:0}: Error finding container 2ae02c18f127dd6840789dec41edc280ebdde74982eb17b1719dc41d02032028: Status 404 returned error can't find the container with id 2ae02c18f127dd6840789dec41edc280ebdde74982eb17b1719dc41d02032028
Oct 31 19:33:04.246584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.246561    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.247531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.247509    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.248810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.248780    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:04.248925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.248842    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:04.248925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.248873    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:04.274944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.274913    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjp9\" (UniqueName: \"kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9\") pod \"maas-api-revoke-sub-2ea630f0-zqkfj\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.354750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.354713    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:04.411560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.410355    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:04.415945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.413416    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:04.415945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.413465    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:04.667442 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:04.667411    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260ea7a5_8029_4c62_9ed5_801c264f7819.slice/crio-0c61d8d1d8c6ed628729692d56efb1d9411dd120c45df50b815e15a9051d0fb1 WatchSource:0}: Error finding container 0c61d8d1d8c6ed628729692d56efb1d9411dd120c45df50b815e15a9051d0fb1: Status 404 returned error can't find the container with id 0c61d8d1d8c6ed628729692d56efb1d9411dd120c45df50b815e15a9051d0fb1
Oct 31 19:33:04.668641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.668619    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"]
Oct 31 19:33:04.942473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.942368    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" event={"ID":"a5f705aa-c316-4b3f-a179-9f65434398c3","Type":"ContainerStarted","Data":"bc0afd3efeff1efdd2e9541c50705130d312246c02cf88162aab96168769cf5c"}
Oct 31 19:33:04.942473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.942462    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" event={"ID":"a5f705aa-c316-4b3f-a179-9f65434398c3","Type":"ContainerStarted","Data":"2ae02c18f127dd6840789dec41edc280ebdde74982eb17b1719dc41d02032028"}
Oct 31 19:33:04.943825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.943801    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj" event={"ID":"260ea7a5-8029-4c62-9ed5-801c264f7819","Type":"ContainerStarted","Data":"0c61d8d1d8c6ed628729692d56efb1d9411dd120c45df50b815e15a9051d0fb1"}
Oct 31 19:33:04.967909 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:04.967852    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podStartSLOduration=1.967836973 podStartE2EDuration="1.967836973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:04.967733576 +0000 UTC m=+778.429108191" watchObservedRunningTime="2026-10-31 19:33:04.967836973 +0000 UTC m=+778.429211588"
Oct 31 19:33:05.413848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.413810    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:05.414438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.414406    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:05.865081 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.865041    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:05.866420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.866384    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:05.866590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.866442    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:05.960942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.960398    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj" event={"ID":"260ea7a5-8029-4c62-9ed5-801c264f7819","Type":"ContainerStarted","Data":"174192f790fb4156510f5d9e18c0b1d26c10d632618e6f2a46d0a8ea537617ff"}
Oct 31 19:33:05.985615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:05.985521    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj" podStartSLOduration=2.9855079030000002 podStartE2EDuration="2.985507903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:05.982075201 +0000 UTC m=+779.443449815" watchObservedRunningTime="2026-10-31 19:33:05.985507903 +0000 UTC m=+779.446882517"
Oct 31 19:33:06.432567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:06.419994    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:06.432567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:06.420058    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:06.864604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:06.864512    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:06.864751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:06.864598    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:06.970641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:06.970604    5958 generic.go:356] "Generic (PLEG): container finished" podID="260ea7a5-8029-4c62-9ed5-801c264f7819" containerID="174192f790fb4156510f5d9e18c0b1d26c10d632618e6f2a46d0a8ea537617ff" exitCode=0
Oct 31 19:33:06.972091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:06.971893    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj" event={"ID":"260ea7a5-8029-4c62-9ed5-801c264f7819","Type":"ContainerDied","Data":"174192f790fb4156510f5d9e18c0b1d26c10d632618e6f2a46d0a8ea537617ff"}
Oct 31 19:33:07.420574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:07.410951    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:07.420574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:07.411029    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:07.865512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:07.865409    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:07.865512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:07.865474    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:08.367498 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.367114    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:08.438865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.438741    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:08.439186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.439134    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:08.574653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.504803    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca\") pod \"260ea7a5-8029-4c62-9ed5-801c264f7819\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") "
Oct 31 19:33:08.574653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.504910    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca\") pod \"260ea7a5-8029-4c62-9ed5-801c264f7819\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") "
Oct 31 19:33:08.574653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.504994    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9\" (UniqueName: \"kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9\") pod \"260ea7a5-8029-4c62-9ed5-801c264f7819\" (UID: \"260ea7a5-8029-4c62-9ed5-801c264f7819\") "
Oct 31 19:33:08.576318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.575598    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca" pod "260ea7a5-8029-4c62-9ed5-801c264f7819" (UID: "260ea7a5-8029-4c62-9ed5-801c264f7819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:33:08.576318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.575801    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca" pod "260ea7a5-8029-4c62-9ed5-801c264f7819" (UID: "260ea7a5-8029-4c62-9ed5-801c264f7819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:08.576746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.576716    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9" pod "260ea7a5-8029-4c62-9ed5-801c264f7819" (UID: "260ea7a5-8029-4c62-9ed5-801c264f7819"). InnerVolumeSpecName "kube-api-access-pnjp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:08.636602 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.616308    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:08.636602 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.616359    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ea7a5-8029-4c62-9ed5-801c264f7819-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:08.636602 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.616375    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjp9\" (UniqueName: \"kubernetes.io/projected/260ea7a5-8029-4c62-9ed5-801c264f7819-kube-api-access-pnjp9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:08.865215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.865129    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:08.865215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.865184    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:08.987090 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.987063    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"
Oct 31 19:33:08.987508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.987092    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj" event={"ID":"260ea7a5-8029-4c62-9ed5-801c264f7819","Type":"ContainerDied","Data":"0c61d8d1d8c6ed628729692d56efb1d9411dd120c45df50b815e15a9051d0fb1"}
Oct 31 19:33:08.987508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:08.987152    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c61d8d1d8c6ed628729692d56efb1d9411dd120c45df50b815e15a9051d0fb1"
Oct 31 19:33:09.437569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:09.423973    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:09.437569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:09.424043    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:09.865149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:09.865059    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:09.865149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:09.865120    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:10.417569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:10.416826    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:10.417569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:10.416900    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:10.864685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:10.864599    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:10.864685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:10.864658    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:11.410676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:11.410634    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 19:33:11.410849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:11.410706    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 19:33:11.866617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:11.865602    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 19:33:11.866617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:11.865682    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 19:33:11.912201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:11.912143    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:33:12.417505 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.417452    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:12.418722 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.418561    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:12.419856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.419821    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 19:33:12.632752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.632718    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"]
Oct 31 19:33:12.633457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.633440    5958 cpu_manager.go:414] "containerMap: removing container" podUID="260ea7a5-8029-4c62-9ed5-801c264f7819" containerName="revoke-keys"
Oct 31 19:33:12.633457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.633454    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260ea7a5-8029-4c62-9ed5-801c264f7819" containerName="revoke-keys"
Oct 31 19:33:12.633669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.633622    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="260ea7a5-8029-4c62-9ed5-801c264f7819" containerName="revoke-keys"
Oct 31 19:33:12.638689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.638665    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.651360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.651335    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"distinct-2-w0-23843f-kserve-self-signed-certs\""
Oct 31 19:33:12.654516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.654488    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"]
Oct 31 19:33:12.684935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.684849    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.684935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.684907    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.685174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.685040    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.685174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.685088    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.685277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.685180    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.685277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.685256    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdj4x\" (UniqueName: \"kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.794568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.793625    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.794568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.793807    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdj4x\" (UniqueName: \"kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.794568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.794570    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.798574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.795073    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.798574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.795419    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.798574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.795458    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.798574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.795591    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.798574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.795885    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.798574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.796073    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.805573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.799892    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.831568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.823538    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.831568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.824467    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdj4x\" (UniqueName: \"kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x\") pod \"distinct-2-w0-23843f-kserve-b6c9c6768-48lgz\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:12.926578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.925989    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:12.931432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.927654    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:12.943630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.937825    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:33:12.972052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:12.971766    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:13.373132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:13.373103    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"]
Oct 31 19:33:13.375736 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:13.375708    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f56a551_769a_4653_bef3_dd83b782d8da.slice/crio-a6d6903c0a73b7238ba6260f52523d171aec8fc308b84e9e762c45efefe6889b WatchSource:0}: Error finding container a6d6903c0a73b7238ba6260f52523d171aec8fc308b84e9e762c45efefe6889b: Status 404 returned error can't find the container with id a6d6903c0a73b7238ba6260f52523d171aec8fc308b84e9e762c45efefe6889b
Oct 31 19:33:14.250567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:14.249663    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" event={"ID":"5f56a551-769a-4653-bef3-dd83b782d8da","Type":"ContainerStarted","Data":"bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e"}
Oct 31 19:33:14.250567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:14.249716    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" event={"ID":"5f56a551-769a-4653-bef3-dd83b782d8da","Type":"ContainerStarted","Data":"a6d6903c0a73b7238ba6260f52523d171aec8fc308b84e9e762c45efefe6889b"}
Oct 31 19:33:14.287620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:14.287212    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podStartSLOduration=2.28719098 podStartE2EDuration="2.28719098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:14.283993279 +0000 UTC m=+787.745367896" watchObservedRunningTime="2026-10-31 19:33:14.28719098 +0000 UTC m=+787.748565596"
Oct 31 19:33:14.905567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:14.889688    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:14.952695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:14.952640    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:14.985573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:14.976623    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:15.084189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.078671    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.103134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.102910    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.172496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.170788    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.430801 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.430667    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.496732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.496678    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:33:15.647582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.647513    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.913075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.913028    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:33:15.913940 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.913915    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:33:15.964623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.964036    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:15.964623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:15.964446    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:33:16.113579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.082763    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.113579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.083415    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:33:16.155581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.131665    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.155581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.132235    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:33:16.272083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.271979    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:16.323329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.323293    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:33:16.648706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.648659    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.661597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.659983    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:33:16.718965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.718912    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:16.855211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.855151    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 31 19:33:16.914685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.913587    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.104645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:16.945872    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 31 19:33:17.228676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.170032    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.278938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.210572    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:17.293186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.211309    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:17.353735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.353688    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.580619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.577627    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.653845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.653780    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.727138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:17.727084    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:18.062599 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:18.042007    5958 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.381s"
Oct 31 19:33:18.062599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.042232    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:33:18.396135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.395806    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:33:18.521859 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:18.521816    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:33:18.977323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.972826    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:33:18.978798 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.978777    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:33:18.979021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.979008    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:33:18.979175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.979165    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:33:18.979339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.979328    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:33:18.979494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:18.979471    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:33:22.974137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:22.974104    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:22.986882 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:22.986856    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:23.223901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:23.223867    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:28.073690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.073658    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"]
Oct 31 19:33:28.078819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.078798    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.084975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.084952    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:33:28.085165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.085075    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:33:28.088198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.088172    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"]
Oct 31 19:33:28.092648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.092446    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9kr\" (UniqueName: \"kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.092779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.092756    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.092851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.092799    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.207613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.197367    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.207613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.197432    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.223205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.223164    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9kr\" (UniqueName: \"kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.244100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.229704    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.244100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.230975    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.281786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.281738    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9kr\" (UniqueName: \"kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr\") pod \"maas-api-revoke-sub-74e78f23-4hxsd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.401238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.401204    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:28.690370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:28.690264    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"]
Oct 31 19:33:28.690493 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:28.690466    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc82231_ff48_493e_bae1_b7ece14675fd.slice/crio-30e4ab421adf116e0a77a0c33a0c01938a0365715b20526daf390df1b85d4ae0 WatchSource:0}: Error finding container 30e4ab421adf116e0a77a0c33a0c01938a0365715b20526daf390df1b85d4ae0: Status 404 returned error can't find the container with id 30e4ab421adf116e0a77a0c33a0c01938a0365715b20526daf390df1b85d4ae0
Oct 31 19:33:29.284078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:29.284038    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd" event={"ID":"3bc82231-ff48-493e-bae1-b7ece14675fd","Type":"ContainerStarted","Data":"db955bfeafde91029ba8af4883825b803264f50c7bfd5ec2110ea3dc4c93a3b0"}
Oct 31 19:33:29.284493 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:29.284158    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd" event={"ID":"3bc82231-ff48-493e-bae1-b7ece14675fd","Type":"ContainerStarted","Data":"30e4ab421adf116e0a77a0c33a0c01938a0365715b20526daf390df1b85d4ae0"}
Oct 31 19:33:29.310398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:29.310347    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd" podStartSLOduration=1.310332472 podStartE2EDuration="1.310332472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:29.304535453 +0000 UTC m=+802.765910068" watchObservedRunningTime="2026-10-31 19:33:29.310332472 +0000 UTC m=+802.771707135"
Oct 31 19:33:30.294241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:30.294190    5958 generic.go:356] "Generic (PLEG): container finished" podID="3bc82231-ff48-493e-bae1-b7ece14675fd" containerID="db955bfeafde91029ba8af4883825b803264f50c7bfd5ec2110ea3dc4c93a3b0" exitCode=0
Oct 31 19:33:30.294776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:30.294277    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd" event={"ID":"3bc82231-ff48-493e-bae1-b7ece14675fd","Type":"ContainerDied","Data":"db955bfeafde91029ba8af4883825b803264f50c7bfd5ec2110ea3dc4c93a3b0"}
Oct 31 19:33:31.482088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.482062    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:31.575783 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.575669    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr\" (UniqueName: \"kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr\") pod \"3bc82231-ff48-493e-bae1-b7ece14675fd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") "
Oct 31 19:33:31.576063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.576044    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca\") pod \"3bc82231-ff48-493e-bae1-b7ece14675fd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") "
Oct 31 19:33:31.576153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.576085    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca\") pod \"3bc82231-ff48-493e-bae1-b7ece14675fd\" (UID: \"3bc82231-ff48-493e-bae1-b7ece14675fd\") "
Oct 31 19:33:31.576467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.576434    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca" pod "3bc82231-ff48-493e-bae1-b7ece14675fd" (UID: "3bc82231-ff48-493e-bae1-b7ece14675fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:31.576703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.576686    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:31.577907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.577878    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr" pod "3bc82231-ff48-493e-bae1-b7ece14675fd" (UID: "3bc82231-ff48-493e-bae1-b7ece14675fd"). InnerVolumeSpecName "kube-api-access-ml9kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:31.578091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.578073    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca" pod "3bc82231-ff48-493e-bae1-b7ece14675fd" (UID: "3bc82231-ff48-493e-bae1-b7ece14675fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:33:31.678355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.678324    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc82231-ff48-493e-bae1-b7ece14675fd-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:31.678355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:31.678352    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9kr\" (UniqueName: \"kubernetes.io/projected/3bc82231-ff48-493e-bae1-b7ece14675fd-kube-api-access-ml9kr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:32.310964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:32.310921    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd" event={"ID":"3bc82231-ff48-493e-bae1-b7ece14675fd","Type":"ContainerDied","Data":"30e4ab421adf116e0a77a0c33a0c01938a0365715b20526daf390df1b85d4ae0"}
Oct 31 19:33:32.311117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:32.310988    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e4ab421adf116e0a77a0c33a0c01938a0365715b20526daf390df1b85d4ae0"
Oct 31 19:33:32.311117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:32.310992    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"
Oct 31 19:33:33.154368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.154327    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"]
Oct 31 19:33:33.155077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.155060    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3bc82231-ff48-493e-bae1-b7ece14675fd" containerName="revoke-keys"
Oct 31 19:33:33.155144 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.155078    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc82231-ff48-493e-bae1-b7ece14675fd" containerName="revoke-keys"
Oct 31 19:33:33.155195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.155186    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc82231-ff48-493e-bae1-b7ece14675fd" containerName="revoke-keys"
Oct 31 19:33:33.159061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.159037    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.186709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.186676    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"]
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207406    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207562    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207650    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrhs\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207685    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207764    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207792    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207845    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207885    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.208247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.207925    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.214107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.214079    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"]
Oct 31 19:33:33.219734 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.219684    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.227471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.227065    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"]
Oct 31 19:33:33.309607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309572    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.309835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309623    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrhs\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.309835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309722    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.309835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309828    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309886    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jx45\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309907    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.309931    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310014    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310048    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310098    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310144    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310265    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310299    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310319    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310375    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310409    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310450    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310472    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310565    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.310743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310627    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.310743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310667    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.311002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310865    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.311002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.310947    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.313063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.313037    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.314176 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.314156    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.338065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.338022    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.353951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.353921    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrhs\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.412019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.411933    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.411974    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412014    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412033    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412071    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412123    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412153    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412502 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412419    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412502 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412492    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jx45\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412529    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412586    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412616    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412841    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.412923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.412869    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.414903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.414877    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.415059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.415042    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.456802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.456767    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.457383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.457362    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jx45\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.487829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.487792    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:33.509753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.509715    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"]
Oct 31 19:33:33.535085 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.535009    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"]
Oct 31 19:33:33.535256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.535158    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.542021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.541791    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:33.624638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624517    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsnh\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.624638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624607    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.624832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624644    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.624832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624762    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.624832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624790    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.624832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624819    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.625031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624859    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.625031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.624942    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.625143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.625059    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830144    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830211    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830241    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830278    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830351    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830446    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.830621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830619    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsnh\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.831139 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830659    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.831139 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.830697    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.916122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.831294    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.916122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.846306    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.916122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.848204    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.916122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.848453    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.916122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.854448    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.947660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.947629    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.962594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.959648    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:33.962594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.960156    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"]
Oct 31 19:33:33.974888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.969198    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:33.974888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.973371    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:33.974888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:33.973426    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:34.053943 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.044918    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:34.155821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.155767    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:34.156324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.156298    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:34.156442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.156426    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:34.162912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.162885    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"]
Oct 31 19:33:34.163118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.163103    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"]
Oct 31 19:33:34.171181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.171146    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsnh\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:34.172399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.172372    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.180746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.177986    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"]
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.389650    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.389714    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.389820    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.389850    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.389893    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.389949    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.390007    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45sw\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.390147    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.396298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.390206    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.466252 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.466216    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.530880    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.530953    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531100    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531141    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531244    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531277    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531318    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531378    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.531568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.531433    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45sw\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.540628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.532617    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.540628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.533278    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.540628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.535371    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.546311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.546267    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.560963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.533536    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.596171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.593880    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" event={"ID":"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4","Type":"ContainerStarted","Data":"b19d8e630b1953014aae9384ac25ea45ff3c4db41c912e9ca4c5a0d343b5e151"}
Oct 31 19:33:34.596171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.596111    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.624626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.624506    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" event={"ID":"74fcf135-c73f-4517-b815-b23a15bf4931","Type":"ContainerStarted","Data":"172f07d90341de0fba376be7bf35c0dc82f8aac61ec8c26d843f5c6e20b828ec"}
Oct 31 19:33:34.758229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.758145    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45sw\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.786675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.786588    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.804750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.800817    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:34.871640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:34.870262    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:35.013138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.011917    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:35.091685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.084965    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:35.289335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.288672    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:35.299807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.298823    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:35.299807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.299203    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:33:35.477564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.463904    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:35.552769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.552662    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:35.959887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.959843    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.240736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.969420    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" event={"ID":"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4","Type":"ContainerStarted","Data":"ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511"}
Oct 31 19:33:36.240736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.231956    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" event={"ID":"74fcf135-c73f-4517-b815-b23a15bf4931","Type":"ContainerStarted","Data":"ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed"}
Oct 31 19:33:36.240736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:35.971608    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:36.240736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.228286    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.240736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.228368    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.240736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.233325    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:33:36.243347 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.081528    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:33:36.311126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.306095    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"]
Oct 31 19:33:36.572620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.523725    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:36.572620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.523797    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:36.606604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.592936    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:36.606604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.593019    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:36.606604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.593645    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.781316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.727083    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:33:36.884166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.813410    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.482871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.831636    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.894745    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.895663    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz" podUID="c104a69f-80bd-47cc-a6f0-b5e6e199e9c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.913886    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:36.989984    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.064343    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.485001    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.485156    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dp8w\" (UniqueName: \"kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.434170    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.263787    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.434113    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.434201    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.487571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.434234    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.639837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.639778    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.799649 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.760130    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.892961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.891778    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:37.892961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.891871    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:37.892961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.891913    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:37.894338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.894292    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:37.894669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.894625    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:37.894883 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.894842    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:33:37.977586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.959608    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dp8w\" (UniqueName: \"kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.977586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.959885    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.977586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.960018    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.977586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:37.961047    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:37.977586 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:37.961510    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/opendatahub-ca: object "odh-ai-gateway-infra"/"opendatahub-ca" not registered
Oct 31 19:33:38.384571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.297981    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dp8w\" (UniqueName: \"kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:38.449255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.449188    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:38.449563 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.449339    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:38.449798 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.449749    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:38.482795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.452638    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:38.483113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.483065    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:38.496921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.487462    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:38.497996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.482686    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:33:38.517890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.517853    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:33:38.517890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.517903    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"]
Oct 31 19:33:38.518149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.517949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:38.518937 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.518822    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:33:38.552815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.551913    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:33:38.552815 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:38.510984    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca podName:d0212e2e-6893-4d65-bfb7-0b410d4e4607 nodeName:}" failed. No retries permitted until 2026-10-31 19:33:39.010956145 +0000 UTC m=+812.472330738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-xks-ca" (UniqueName: "kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca") pod "maas-api-revoke-sub-53342951-hz6rh" (UID: "d0212e2e-6893-4d65-bfb7-0b410d4e4607") : object "odh-ai-gateway-infra"/"opendatahub-ca" not registered
Oct 31 19:33:38.566198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.537096    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:38.575107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.575064    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:38.674153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.674036    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podStartSLOduration=5.674012834 podStartE2EDuration="5.674012834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:36.08008256 +0000 UTC m=+809.541457176" watchObservedRunningTime="2026-10-31 19:33:38.674012834 +0000 UTC m=+812.135387450"
Oct 31 19:33:38.793682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.793646    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:38.847574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:38.843069    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:39.117795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.101638    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:39.406567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.406201    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podStartSLOduration=6.406170455 podStartE2EDuration="6.406170455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:36.415213281 +0000 UTC m=+809.876587898" watchObservedRunningTime="2026-10-31 19:33:39.406170455 +0000 UTC m=+812.867545070"
Oct 31 19:33:39.450581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.444139    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.450581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.445608    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.450581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.446919    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.450581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.447597    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.488186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.483502    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.488186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.483857    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.488186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.484086    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.488186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.484273    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.681663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.612903    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:39.681663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.612962    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:39.681663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.621074    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm" podUID="75c5a1ce-03a1-44dd-a80a-72e6addba5f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.729712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.723781    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 31 19:33:39.743752 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:39.743708    5958 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.226s"
Oct 31 19:33:39.818571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.797233    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.937622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:39.920331    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53342951-hz6rh\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:40.038677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.027957    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:40.038677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.028016    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:40.131182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.129152    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" event={"ID":"fc4a7691-ea30-4bca-bc1b-0b195ef17a00","Type":"ContainerStarted","Data":"4b1cf336ae5255404cdfd507c8e38a9a565820ec3fdbbbe17c8371edc9932407"}
Oct 31 19:33:40.131182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.129273    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:40.131182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.129376    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:33:40.131182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.129416    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:33:40.131182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.129458    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:33:40.131182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.129497    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:33:40.186595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.176036    5958 trace.go:236] Trace[1142354019]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-dj7j4" (31-Oct-2026 19:33:38.726) (total time: 1449ms):
Oct 31 19:33:40.186595 ip-10-0-138-35 kubenswrapper[5958]: Trace[1142354019]: [1.449156758s] [1.449156758s] END
Oct 31 19:33:40.186595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.176365    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:33:40.186595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.176712    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:40.235201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.235170    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:40.388021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.329271    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:33:40.388021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.329731    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:33:40.410602 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.399674    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:40.716286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.631807    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:40.716286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.631890    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:40.716286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.673529    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:40.716286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.673603    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:40.744175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.744128    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:33:40.744675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.744461    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:40.749596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.744820    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:33:40.749596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.745021    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:33:40.997704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:40.993833    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:41.059818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.059769    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:41.179720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.179688    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:33:41.180020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.180002    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:33:41.180157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.180142    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"]
Oct 31 19:33:41.705098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.705057    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"]
Oct 31 19:33:41.705484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.705463    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:33:41.705739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.705723    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:33:41.750164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.708690    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:33:41.770647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.757025    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:33:41.802964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.802915    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:33:41.951569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:41.950721    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:42.065439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.065402    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" event={"ID":"1655e665-7376-41cf-b2ab-44a2d1405bad","Type":"ContainerStarted","Data":"133df5c14cf6f94679ab442759aabbb5a15600204da491c16d4fa6fc7eabdce1"}
Oct 31 19:33:42.078475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.078442    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.078765 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.078735    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:42.098520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.098477    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.111782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.099627    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:42.498621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.497756    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:42.501218 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.500988    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:42.553624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.541152    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.553624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.541232    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:42.553624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.544285    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.553624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.544342    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:42.592715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.592669    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.592983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.592953    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:42.695578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:42.695192    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:43.376753 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:43.355379    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache]"
Oct 31 19:33:43.521787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.521748    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.522088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.522055    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:43.523070 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.523047    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.523240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.523203    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:43.558804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.558767    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.571789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.568079    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:43.686630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.669804    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" event={"ID":"fc4a7691-ea30-4bca-bc1b-0b195ef17a00","Type":"ContainerStarted","Data":"637dcbba0a35eceaeb3e4b57cfdf6b216f21f82335ede4f82189cb740e151090"}
Oct 31 19:33:43.695667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.695618    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:33:43.785567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.770385    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:43.785567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.779752    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:33:43.871570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.840495    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" event={"ID":"1655e665-7376-41cf-b2ab-44a2d1405bad","Type":"ContainerStarted","Data":"4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e"}
Oct 31 19:33:43.892808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.873856    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh" event={"ID":"d0212e2e-6893-4d65-bfb7-0b410d4e4607","Type":"ContainerStarted","Data":"32b6f6d056ca4b519412c030b2089210e18b6beb984668334287178f42d0bf81"}
Oct 31 19:33:43.929475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.900616    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:43.929775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.901036    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.929968 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.929941    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:43.990753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:43.990642    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.061695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.061645    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:44.182885 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.182810    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.240588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.238689    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.450141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.450093    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.485694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.470299    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.485694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.470358    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:44.485694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.478701    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:33:44.502705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.502647    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.561577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.561229    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.561577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.561302    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:44.562491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.562439    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.602161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.602111    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:44.709633 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.708798    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded"
Oct 31 19:33:44.860613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.834449    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2fbkm" podUID="75c5a1ce-03a1-44dd-a80a-72e6addba5f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.860613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.837166    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:33:44.860613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.858677    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:44.891135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.891098    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.908850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:44.908811    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:45.097626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.066979    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:45.250569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.245851    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:45.326568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.297119    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:45.326833 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.326803    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh" event={"ID":"d0212e2e-6893-4d65-bfb7-0b410d4e4607","Type":"ContainerStarted","Data":"b803c184b1cf153d941346d1bd9eaddae151ee1950f8e9ebc0853fbd3694421a"}
Oct 31 19:33:45.327011 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.326992    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"]
Oct 31 19:33:45.327190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.327176    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.516522    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.516638    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.522627    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.470230    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.465099    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.317670    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.470173    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:45.620010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.611194    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.655822    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.763777    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.765369    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.765509    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.765595    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.817858    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.817934    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.867291    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.893249    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.893312    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.934656    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:45.936444    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.010000    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.031420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.010054    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:46.476567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.370374    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:33:46.526629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.526524    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podStartSLOduration=13.526499991 podStartE2EDuration="13.526499991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:42.600478127 +0000 UTC m=+816.061852746" watchObservedRunningTime="2026-10-31 19:33:46.526499991 +0000 UTC m=+819.987874608"
Oct 31 19:33:46.593860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.552485    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.630100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.613466    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:46.630100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.612276    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:46.630100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.619630    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.630100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.619688    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:46.630100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.629687    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.630100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.629760    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:46.655536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.654292    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:33:46.691587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.691533    5958 generic.go:356] "Generic (PLEG): container finished" podID="d0212e2e-6893-4d65-bfb7-0b410d4e4607" containerID="b803c184b1cf153d941346d1bd9eaddae151ee1950f8e9ebc0853fbd3694421a" exitCode=0
Oct 31 19:33:46.697587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.692697    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh" event={"ID":"d0212e2e-6893-4d65-bfb7-0b410d4e4607","Type":"ContainerDied","Data":"b803c184b1cf153d941346d1bd9eaddae151ee1950f8e9ebc0853fbd3694421a"}
Oct 31 19:33:46.745767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.713923    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:33:46.781571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.778734    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:46.781571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.780014    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:46.941568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:46.936768    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:47.019119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.018816    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Oct 31 19:33:47.020063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.020018    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.086446 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.086404    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:47.167574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.165098    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:47.167574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.155083    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.167574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.105999    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.204468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.204421    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.448624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.348456    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podStartSLOduration=14.34843524 podStartE2EDuration="14.34843524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:43.899287776 +0000 UTC m=+817.360662394" watchObservedRunningTime="2026-10-31 19:33:47.34843524 +0000 UTC m=+820.809809855"
Oct 31 19:33:47.499569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.495646    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:47.499569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.495711    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:47.538618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.536209    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.538618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.538585    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:33:47.539083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.539062    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:33:47.540978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.540842    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:33:47.541439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.541417    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 19:33:47.542288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.542246    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 19:33:47.542502 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.542482    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 19:33:47.558573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.549501    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh" podStartSLOduration=12.549483834 podStartE2EDuration="12.549483834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:45.405225868 +0000 UTC m=+818.866600485" watchObservedRunningTime="2026-10-31 19:33:47.549483834 +0000 UTC m=+821.010858453"
Oct 31 19:33:47.587168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.584840    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.600858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.600826    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:33:47.648570 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:33:47.643804    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:33:47.648570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.628338    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:47.686581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.675168    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:33:47.705107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.705026    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:33:47.826570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.817744    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:33:47.945877 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.940274    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:47.945877 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:47.940329    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:48.029677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.021246    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:33:48.082797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.081594    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:33:48.090199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.090160    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:33:48.494862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.493076    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:48.708569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.706976    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:48.708569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.707061    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:48.895868 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.895820    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:48.896283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.896252    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:48.901304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:48.901278    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:49.001890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.001852    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:49.096074 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.096022    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:49.572629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.571859    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:49.572629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.571936    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:49.597513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.597476    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:49.597814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.597780    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:49.601996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.601971    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:33:49.640928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.640885    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:33:49.694102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.694059    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:49.767264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.767238    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"]
Oct 31 19:33:49.770935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.770898    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d0212e2e-6893-4d65-bfb7-0b410d4e4607" containerName="revoke-keys"
Oct 31 19:33:49.773767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.773742    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0212e2e-6893-4d65-bfb7-0b410d4e4607" containerName="revoke-keys"
Oct 31 19:33:49.774252 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.774232    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0212e2e-6893-4d65-bfb7-0b410d4e4607" containerName="revoke-keys"
Oct 31 19:33:49.776514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.772997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca\") pod \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") "
Oct 31 19:33:49.776816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.776797    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\") pod \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") "
Oct 31 19:33:49.777057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.777041    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w\" (UniqueName: \"kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w\") pod \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\" (UID: \"d0212e2e-6893-4d65-bfb7-0b410d4e4607\") "
Oct 31 19:33:49.778533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.775201    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca" pod "d0212e2e-6893-4d65-bfb7-0b410d4e4607" (UID: "d0212e2e-6893-4d65-bfb7-0b410d4e4607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:49.803820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.800355    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"]
Oct 31 19:33:49.803820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.800497    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.810392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.810203    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:49.814305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.814273    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca" pod "d0212e2e-6893-4d65-bfb7-0b410d4e4607" (UID: "d0212e2e-6893-4d65-bfb7-0b410d4e4607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:33:49.845889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.845810    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w" pod "d0212e2e-6893-4d65-bfb7-0b410d4e4607" (UID: "d0212e2e-6893-4d65-bfb7-0b410d4e4607"). InnerVolumeSpecName "kube-api-access-9dp8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:49.884858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.884800    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.885127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.885107    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.885309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.885292    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44h6c\" (UniqueName: \"kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.885736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.885715    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dp8w\" (UniqueName: \"kubernetes.io/projected/d0212e2e-6893-4d65-bfb7-0b410d4e4607-kube-api-access-9dp8w\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:49.885860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.885848    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:49.885946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.885935    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0212e2e-6893-4d65-bfb7-0b410d4e4607-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:49.886410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.886387    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:49.886591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.886567    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:49.887076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.887058    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:49.953340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.953304    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"
Oct 31 19:33:49.953340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.953318    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh" event={"ID":"d0212e2e-6893-4d65-bfb7-0b410d4e4607","Type":"ContainerDied","Data":"32b6f6d056ca4b519412c030b2089210e18b6beb984668334287178f42d0bf81"}
Oct 31 19:33:49.953600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.953408    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b6f6d056ca4b519412c030b2089210e18b6beb984668334287178f42d0bf81"
Oct 31 19:33:49.955209 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.955183    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:33:49.990262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.990224    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.990885 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.990853    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.991610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.991586    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44h6c\" (UniqueName: \"kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.997066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.992481    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:49.997066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:49.995677    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:50.029362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.029145    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44h6c\" (UniqueName: \"kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c\") pod \"maas-api-revoke-sub-72c59481-8pp6q\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:50.164169 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.164123    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:50.535601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.533728    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 19:33:50.535601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.533788    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 19:33:50.550567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.548750    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"]
Oct 31 19:33:50.560376 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:50.558798    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e99a89_3c6b_423e_8b69_604048be0f79.slice/crio-3b8dd3a74c411b39ff79497cef53e06eae2f395fceffa6c358d195a90e7db203 WatchSource:0}: Error finding container 3b8dd3a74c411b39ff79497cef53e06eae2f395fceffa6c358d195a90e7db203: Status 404 returned error can't find the container with id 3b8dd3a74c411b39ff79497cef53e06eae2f395fceffa6c358d195a90e7db203
Oct 31 19:33:50.882738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.882695    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:33:50.883258 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.882765    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 19:33:50.965948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:50.965911    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q" event={"ID":"f9e99a89-3c6b-423e-8b69-604048be0f79","Type":"ContainerStarted","Data":"3b8dd3a74c411b39ff79497cef53e06eae2f395fceffa6c358d195a90e7db203"}
Oct 31 19:33:51.556572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.552530    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:51.556842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.556777    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:33:51.596836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.596799    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"]
Oct 31 19:33:51.605361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.603918    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.621328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.621298    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"]
Oct 31 19:33:51.778186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.778142    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.778432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.778401    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrkn\" (UniqueName: \"kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.778492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.778475    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.880396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.880351    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.880589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.880526    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrkn\" (UniqueName: \"kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.880754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.880660    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.881332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.881299    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.884181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.883828    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.887706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.887669    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:51.888956 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.888936    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:33:51.898747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.898722    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrkn\" (UniqueName: \"kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn\") pod \"maas-api-revoke-sub-fa1fc859-zmvmj\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:51.956895 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:51.956859    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:52.002618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:52.002578    5958 generic.go:356] "Generic (PLEG): container finished" podID="f9e99a89-3c6b-423e-8b69-604048be0f79" containerID="c8da6ebf76d968f33223efd1b826978f032778c6fdf4daa30ec62345d9794065" exitCode=0
Oct 31 19:33:52.004310 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:52.004270    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q" event={"ID":"f9e99a89-3c6b-423e-8b69-604048be0f79","Type":"ContainerDied","Data":"c8da6ebf76d968f33223efd1b826978f032778c6fdf4daa30ec62345d9794065"}
Oct 31 19:33:52.297928 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:33:52.297897    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e42f63_a3ae_4d6b_b868_86dfba23ffe3.slice/crio-5712eb96d66aed50a0c6afb534d7747bceb64c4308ce80fa7ac869d03085ae6a WatchSource:0}: Error finding container 5712eb96d66aed50a0c6afb534d7747bceb64c4308ce80fa7ac869d03085ae6a: Status 404 returned error can't find the container with id 5712eb96d66aed50a0c6afb534d7747bceb64c4308ce80fa7ac869d03085ae6a
Oct 31 19:33:52.299118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:52.299091    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"]
Oct 31 19:33:53.017891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.017790    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj" event={"ID":"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3","Type":"ContainerStarted","Data":"b2b7c0ec29b03e7dd5f5250feed139d0dbda294266cdfb16b7f26605f7a19d5f"}
Oct 31 19:33:53.017891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.017856    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj" event={"ID":"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3","Type":"ContainerStarted","Data":"5712eb96d66aed50a0c6afb534d7747bceb64c4308ce80fa7ac869d03085ae6a"}
Oct 31 19:33:53.050904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.050843    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj" podStartSLOduration=2.050826526 podStartE2EDuration="2.050826526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:53.048992094 +0000 UTC m=+826.510366711" watchObservedRunningTime="2026-10-31 19:33:53.050826526 +0000 UTC m=+826.512201142"
Oct 31 19:33:53.255810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.255784    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:53.263629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.263600    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca\") pod \"f9e99a89-3c6b-423e-8b69-604048be0f79\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") "
Oct 31 19:33:53.263784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.263709    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c\" (UniqueName: \"kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c\") pod \"f9e99a89-3c6b-423e-8b69-604048be0f79\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") "
Oct 31 19:33:53.263860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.263848    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca\") pod \"f9e99a89-3c6b-423e-8b69-604048be0f79\" (UID: \"f9e99a89-3c6b-423e-8b69-604048be0f79\") "
Oct 31 19:33:53.264225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.264196    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca" pod "f9e99a89-3c6b-423e-8b69-604048be0f79" (UID: "f9e99a89-3c6b-423e-8b69-604048be0f79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:53.264596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.264535    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:53.265879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.265857    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca" pod "f9e99a89-3c6b-423e-8b69-604048be0f79" (UID: "f9e99a89-3c6b-423e-8b69-604048be0f79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:33:53.265964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.265859    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c" pod "f9e99a89-3c6b-423e-8b69-604048be0f79" (UID: "f9e99a89-3c6b-423e-8b69-604048be0f79"). InnerVolumeSpecName "kube-api-access-44h6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:53.379121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.378281    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44h6c\" (UniqueName: \"kubernetes.io/projected/f9e99a89-3c6b-423e-8b69-604048be0f79-kube-api-access-44h6c\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:53.379121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:53.378324    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e99a89-3c6b-423e-8b69-604048be0f79-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:54.027992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:54.027904    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"
Oct 31 19:33:54.027992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:54.027924    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q" event={"ID":"f9e99a89-3c6b-423e-8b69-604048be0f79","Type":"ContainerDied","Data":"3b8dd3a74c411b39ff79497cef53e06eae2f395fceffa6c358d195a90e7db203"}
Oct 31 19:33:54.027992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:54.027978    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8dd3a74c411b39ff79497cef53e06eae2f395fceffa6c358d195a90e7db203"
Oct 31 19:33:54.030244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:54.030214    5958 generic.go:356] "Generic (PLEG): container finished" podID="f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" containerID="b2b7c0ec29b03e7dd5f5250feed139d0dbda294266cdfb16b7f26605f7a19d5f" exitCode=0
Oct 31 19:33:54.030408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:54.030350    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj" event={"ID":"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3","Type":"ContainerDied","Data":"b2b7c0ec29b03e7dd5f5250feed139d0dbda294266cdfb16b7f26605f7a19d5f"}
Oct 31 19:33:55.397677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.397650    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:55.447771 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.447740    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn\" (UniqueName: \"kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn\") pod \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") "
Oct 31 19:33:55.447928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.447793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca\") pod \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") "
Oct 31 19:33:55.447928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.447890    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca\") pod \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\" (UID: \"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3\") "
Oct 31 19:33:55.448240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.448213    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca" pod "f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" (UID: "f4e42f63-a3ae-4d6b-b868-86dfba23ffe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:33:55.448701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.448670    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:55.450188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.450151    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn" pod "f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" (UID: "f4e42f63-a3ae-4d6b-b868-86dfba23ffe3"). InnerVolumeSpecName "kube-api-access-9vrkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:33:55.450288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.450219    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca" pod "f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" (UID: "f4e42f63-a3ae-4d6b-b868-86dfba23ffe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:33:55.550054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.550017    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrkn\" (UniqueName: \"kubernetes.io/projected/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-kube-api-access-9vrkn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:55.550054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:55.550051    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:33:56.124564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:56.116236    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj" event={"ID":"f4e42f63-a3ae-4d6b-b868-86dfba23ffe3","Type":"ContainerDied","Data":"5712eb96d66aed50a0c6afb534d7747bceb64c4308ce80fa7ac869d03085ae6a"}
Oct 31 19:33:56.124564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:56.116336    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5712eb96d66aed50a0c6afb534d7747bceb64c4308ce80fa7ac869d03085ae6a"
Oct 31 19:33:56.124564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:56.116410    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"
Oct 31 19:33:59.201786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.201752    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"]
Oct 31 19:33:59.202504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.202490    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f9e99a89-3c6b-423e-8b69-604048be0f79" containerName="revoke-keys"
Oct 31 19:33:59.202569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.202506    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e99a89-3c6b-423e-8b69-604048be0f79" containerName="revoke-keys"
Oct 31 19:33:59.202569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.202528    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" containerName="revoke-keys"
Oct 31 19:33:59.202569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.202533    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" containerName="revoke-keys"
Oct 31 19:33:59.202684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.202649    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e99a89-3c6b-423e-8b69-604048be0f79" containerName="revoke-keys"
Oct 31 19:33:59.202684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.202661    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" containerName="revoke-keys"
Oct 31 19:33:59.206777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.206753    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.219761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.219724    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"]
Oct 31 19:33:59.249265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.249227    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:33:59.249433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.249319    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:33:59.279454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.279419    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.279454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.279460    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.279785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.279616    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mwx\" (UniqueName: \"kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.381846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.381808    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.381846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.381851    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.382051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.381895    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mwx\" (UniqueName: \"kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.382488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.382458    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.384291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.384267    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.397984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.397955    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mwx\" (UniqueName: \"kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx\") pod \"maas-api-revoke-sub-698f142d-wgdc9\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.540414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.540324    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:33:59.889606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:33:59.880929    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"]
Oct 31 19:34:00.152268 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:00.152171    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9" event={"ID":"f9890e04-0403-4496-abf7-4d19d34acb42","Type":"ContainerStarted","Data":"31cefb53240cc0476e146f2bcec5bc117128392d5130753c217c8b82785df696"}
Oct 31 19:34:01.161195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:01.161162    5958 generic.go:356] "Generic (PLEG): container finished" podID="f9890e04-0403-4496-abf7-4d19d34acb42" containerID="225ccfd3752ce9ce95de152d7eadaab8e38f64a772e28d8f158d988cd6937e2a" exitCode=0
Oct 31 19:34:01.161651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:01.161238    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9" event={"ID":"f9890e04-0403-4496-abf7-4d19d34acb42","Type":"ContainerDied","Data":"225ccfd3752ce9ce95de152d7eadaab8e38f64a772e28d8f158d988cd6937e2a"}
Oct 31 19:34:02.343291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.343265    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:34:02.438615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.435398    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx\" (UniqueName: \"kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx\") pod \"f9890e04-0403-4496-abf7-4d19d34acb42\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") "
Oct 31 19:34:02.438615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.435695    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca\") pod \"f9890e04-0403-4496-abf7-4d19d34acb42\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") "
Oct 31 19:34:02.438615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.435821    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca\") pod \"f9890e04-0403-4496-abf7-4d19d34acb42\" (UID: \"f9890e04-0403-4496-abf7-4d19d34acb42\") "
Oct 31 19:34:02.456746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.447610    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca" pod "f9890e04-0403-4496-abf7-4d19d34acb42" (UID: "f9890e04-0403-4496-abf7-4d19d34acb42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:34:02.456746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.448755    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:02.475599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.474697    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca" pod "f9890e04-0403-4496-abf7-4d19d34acb42" (UID: "f9890e04-0403-4496-abf7-4d19d34acb42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:34:02.547999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.547957    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx" pod "f9890e04-0403-4496-abf7-4d19d34acb42" (UID: "f9890e04-0403-4496-abf7-4d19d34acb42"). InnerVolumeSpecName "kube-api-access-b4mwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:02.550775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.550634    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9890e04-0403-4496-abf7-4d19d34acb42-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:02.550775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:02.550654    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mwx\" (UniqueName: \"kubernetes.io/projected/f9890e04-0403-4496-abf7-4d19d34acb42-kube-api-access-b4mwx\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:03.179008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:03.178971    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"
Oct 31 19:34:03.179008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:03.178996    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9" event={"ID":"f9890e04-0403-4496-abf7-4d19d34acb42","Type":"ContainerDied","Data":"31cefb53240cc0476e146f2bcec5bc117128392d5130753c217c8b82785df696"}
Oct 31 19:34:03.179222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:03.179050    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31cefb53240cc0476e146f2bcec5bc117128392d5130753c217c8b82785df696"
Oct 31 19:34:08.328840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.328803    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:34:08.846240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.846204    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"]
Oct 31 19:34:08.847005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.846986    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f9890e04-0403-4496-abf7-4d19d34acb42" containerName="revoke-keys"
Oct 31 19:34:08.847005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.847003    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9890e04-0403-4496-abf7-4d19d34acb42" containerName="revoke-keys"
Oct 31 19:34:08.847176 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.847109    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9890e04-0403-4496-abf7-4d19d34acb42" containerName="revoke-keys"
Oct 31 19:34:08.851054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.851036    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:08.857586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.857560    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:34:08.857705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.857657    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:34:08.859203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.859182    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"]
Oct 31 19:34:08.964576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.960409    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hjf\" (UniqueName: \"kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:08.964576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.960709    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:08.964576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:08.960779    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.062756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.062729    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hjf\" (UniqueName: \"kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.063253 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.063206    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.063387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.063272    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.064345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.064322    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.066669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.066644    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.078457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.078422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hjf\" (UniqueName: \"kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf\") pod \"maas-api-revoke-sub-67129660-b9tmd\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.172185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.172151    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:09.488314 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:09.488277    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc928f55f_fdf6_4616_8559_964634333fca.slice/crio-59366c1b2b5fb806233ff4532ee23bc2eb3f4669a8dd95dc06ca2fbfa3dacff1 WatchSource:0}: Error finding container 59366c1b2b5fb806233ff4532ee23bc2eb3f4669a8dd95dc06ca2fbfa3dacff1: Status 404 returned error can't find the container with id 59366c1b2b5fb806233ff4532ee23bc2eb3f4669a8dd95dc06ca2fbfa3dacff1
Oct 31 19:34:09.489706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:09.489672    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"]
Oct 31 19:34:10.417620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.417575    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd" event={"ID":"c928f55f-fdf6-4616-8559-964634333fca","Type":"ContainerStarted","Data":"59366c1b2b5fb806233ff4532ee23bc2eb3f4669a8dd95dc06ca2fbfa3dacff1"}
Oct 31 19:34:10.702980 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.676641    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:10.702980 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.702580    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:10.714847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.714769    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:10.797568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.763702    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:10.799391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.798923    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:10.871705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:10.871646    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.020781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.020674    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.042684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.042635    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.111278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.111217    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:11.523592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.514244    5958 generic.go:356] "Generic (PLEG): container finished" podID="c928f55f-fdf6-4616-8559-964634333fca" containerID="b6a55ebe0df9a5b12ebfe888544fc646881faba43f9289c3ec4d10134a83109a" exitCode=0
Oct 31 19:34:11.523592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.514481    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd" event={"ID":"c928f55f-fdf6-4616-8559-964634333fca","Type":"ContainerDied","Data":"b6a55ebe0df9a5b12ebfe888544fc646881faba43f9289c3ec4d10134a83109a"}
Oct 31 19:34:11.702702 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.702648    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:11.715085 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.715049    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:34:11.769598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.743916    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.963079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.769881    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.963079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.962300    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:34:11.963079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.962417    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:34:11.963079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.795626    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.970574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.805661    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.970574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.901390    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.975573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.970925    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:34:11.975573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.970994    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:34:11.975573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:11.971031    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:34:12.043048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.042994    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.090288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.086619    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:34:12.090932 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.090908    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:34:12.091149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.091122    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:34:12.130815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.113690    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.131650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.131619    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:34:12.236630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.214881    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.709821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.708334    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.759266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.758352    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:12.759266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.758428    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.877607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.864853    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:12.877607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.864920    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.880723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.880679    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:12.880972 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.880940    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.999625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.976941    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:34:12.999625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.986695    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:34:12.999625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.987782    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.999625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:12.996277    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.418598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.418535    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.419288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.419255    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.459068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.459024    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.523026 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.522981    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.551913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.551872    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:13.552185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.552151    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.718195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.718078    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.741571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.721677    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:13.741571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.721743    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.839468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.839423    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:13.840051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:13.840016    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.021068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.020962    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.030767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.024513    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.038372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.024567    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded"
Oct 31 19:34:14.038747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.024615    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.043063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.024639    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.160750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.148214    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:14.160750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.148273    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.419196 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.419150    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.454399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.454352    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.465868 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.465694    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.903716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.763724    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:14.903716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.764344    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.935043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.934999    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.981579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:14.938730    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.164048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.163930    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.439113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.438920    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:34:15.633455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.627257    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:34:15.641486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.641346    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:15.668597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.661738    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:34:15.889111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.874651    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.923296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.914267    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.923296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:15.914758    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.55:50052\" within 1s: context deadline exceeded"
Oct 31 19:34:16.059446 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.059353    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:16.164286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.144380    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6" podUID="3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:16.165454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.165427    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:16.193500 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:34:16.193445    5958 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.028s"
Oct 31 19:34:16.193781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.193758    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:34:16.193990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.193978    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:34:16.194166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.194155    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:34:16.194347 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.194336    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:34:16.194452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.194443    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:34:16.194642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.194533    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:34:16.196461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.196441    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:34:16.200759 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:16.200740    5958 logging.go:55] [core] [Channel #1112 SubChannel #1113]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.55:50052", ServerName: "10.134.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:58694->10.134.0.55:50052: use of closed network connection"
Oct 31 19:34:16.205608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.205587    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:34:16.206872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.206846    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:34:16.260978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.260949    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf\" (UniqueName: \"kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf\") pod \"c928f55f-fdf6-4616-8559-964634333fca\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") "
Oct 31 19:34:16.261143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.261019    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca\") pod \"c928f55f-fdf6-4616-8559-964634333fca\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") "
Oct 31 19:34:16.261283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.261258    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca\") pod \"c928f55f-fdf6-4616-8559-964634333fca\" (UID: \"c928f55f-fdf6-4616-8559-964634333fca\") "
Oct 31 19:34:16.261667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.261635    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca" pod "c928f55f-fdf6-4616-8559-964634333fca" (UID: "c928f55f-fdf6-4616-8559-964634333fca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:34:16.262562 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.262519    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c928f55f-fdf6-4616-8559-964634333fca-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:16.263359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.263337    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf" pod "c928f55f-fdf6-4616-8559-964634333fca" (UID: "c928f55f-fdf6-4616-8559-964634333fca"). InnerVolumeSpecName "kube-api-access-84hjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:16.263618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.263598    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca" pod "c928f55f-fdf6-4616-8559-964634333fca" (UID: "c928f55f-fdf6-4616-8559-964634333fca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:34:16.365431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.365254    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hjf\" (UniqueName: \"kubernetes.io/projected/c928f55f-fdf6-4616-8559-964634333fca-kube-api-access-84hjf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:16.365431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.365293    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c928f55f-fdf6-4616-8559-964634333fca-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:16.377458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.377419    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"]
Oct 31 19:34:16.378911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.378878    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c928f55f-fdf6-4616-8559-964634333fca" containerName="revoke-keys"
Oct 31 19:34:16.378911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.378909    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c928f55f-fdf6-4616-8559-964634333fca" containerName="revoke-keys"
Oct 31 19:34:16.379122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.379114    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c928f55f-fdf6-4616-8559-964634333fca" containerName="revoke-keys"
Oct 31 19:34:16.385787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.385506    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.392072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.392039    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"]
Oct 31 19:34:16.467266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.467230    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.467530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.467515    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.467699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.467670    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpbv\" (UniqueName: \"kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.576182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.576146    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.576389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.576240    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpbv\" (UniqueName: \"kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.576389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.576304    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.577188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.577153    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.579037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.579013    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.590976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.590940    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpbv\" (UniqueName: \"kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv\") pod \"maas-api-revoke-sub-f32bd03c-tk5lf\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:16.702371 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:16.702332    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:17.048483 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:34:17.048395    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:34:17.113186 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:17.113152    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66c20c4_6181_4c46_861a_490262869859.slice/crio-76feea81fe9fd332a92662f31bdde0a95814716440379f6885d4134a26f988b8 WatchSource:0}: Error finding container 76feea81fe9fd332a92662f31bdde0a95814716440379f6885d4134a26f988b8: Status 404 returned error can't find the container with id 76feea81fe9fd332a92662f31bdde0a95814716440379f6885d4134a26f988b8
Oct 31 19:34:17.114393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:17.114366    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"]
Oct 31 19:34:17.135538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:17.135504    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd" event={"ID":"c928f55f-fdf6-4616-8559-964634333fca","Type":"ContainerDied","Data":"59366c1b2b5fb806233ff4532ee23bc2eb3f4669a8dd95dc06ca2fbfa3dacff1"}
Oct 31 19:34:17.135538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:17.135532    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"
Oct 31 19:34:17.135538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:17.135580    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59366c1b2b5fb806233ff4532ee23bc2eb3f4669a8dd95dc06ca2fbfa3dacff1"
Oct 31 19:34:18.146154 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:18.146116    5958 generic.go:356] "Generic (PLEG): container finished" podID="f66c20c4-6181-4c46-861a-490262869859" containerID="63c9a08723010ac85b7be05523c764262b2fe1c0eadd10aeeb78d71a1ec50dee" exitCode=0
Oct 31 19:34:18.146573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:18.146205    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf" event={"ID":"f66c20c4-6181-4c46-861a-490262869859","Type":"ContainerDied","Data":"63c9a08723010ac85b7be05523c764262b2fe1c0eadd10aeeb78d71a1ec50dee"}
Oct 31 19:34:18.146573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:18.146273    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf" event={"ID":"f66c20c4-6181-4c46-861a-490262869859","Type":"ContainerStarted","Data":"76feea81fe9fd332a92662f31bdde0a95814716440379f6885d4134a26f988b8"}
Oct 31 19:34:19.339751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.339726    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:19.360916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.360457    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca\") pod \"f66c20c4-6181-4c46-861a-490262869859\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") "
Oct 31 19:34:19.361131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.361089    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv\" (UniqueName: \"kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv\") pod \"f66c20c4-6181-4c46-861a-490262869859\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") "
Oct 31 19:34:19.361232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.361185    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca\") pod \"f66c20c4-6181-4c46-861a-490262869859\" (UID: \"f66c20c4-6181-4c46-861a-490262869859\") "
Oct 31 19:34:19.364701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.362370    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca" pod "f66c20c4-6181-4c46-861a-490262869859" (UID: "f66c20c4-6181-4c46-861a-490262869859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:34:19.367261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.367225    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca" pod "f66c20c4-6181-4c46-861a-490262869859" (UID: "f66c20c4-6181-4c46-861a-490262869859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:34:19.367411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.367366    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv" pod "f66c20c4-6181-4c46-861a-490262869859" (UID: "f66c20c4-6181-4c46-861a-490262869859"). InnerVolumeSpecName "kube-api-access-nlpbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:19.463955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.463874    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpbv\" (UniqueName: \"kubernetes.io/projected/f66c20c4-6181-4c46-861a-490262869859-kube-api-access-nlpbv\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:19.463955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.463905    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66c20c4-6181-4c46-861a-490262869859-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:19.463955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:19.463916    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66c20c4-6181-4c46-861a-490262869859-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:20.163869 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:20.163841    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"
Oct 31 19:34:20.164022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:20.163875    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf" event={"ID":"f66c20c4-6181-4c46-861a-490262869859","Type":"ContainerDied","Data":"76feea81fe9fd332a92662f31bdde0a95814716440379f6885d4134a26f988b8"}
Oct 31 19:34:20.164022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:20.163950    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76feea81fe9fd332a92662f31bdde0a95814716440379f6885d4134a26f988b8"
Oct 31 19:34:21.470487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.466919    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"]
Oct 31 19:34:21.470487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.468044    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f66c20c4-6181-4c46-861a-490262869859" containerName="revoke-keys"
Oct 31 19:34:21.470487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.468065    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66c20c4-6181-4c46-861a-490262869859" containerName="revoke-keys"
Oct 31 19:34:21.470487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.468244    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66c20c4-6181-4c46-861a-490262869859" containerName="revoke-keys"
Oct 31 19:34:21.486391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.484832    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"]
Oct 31 19:34:21.486391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.484954    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.500211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.500182    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:34:21.500840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.500813    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:34:21.513390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.513355    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmh7s\" (UniqueName: \"kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.513608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.513586    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.513715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.513681    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.615473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.615440    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmh7s\" (UniqueName: \"kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.615665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.615610    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.615737 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.615703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.616280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.616254    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.618181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.618149    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.637657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.637622    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmh7s\" (UniqueName: \"kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s\") pod \"maas-api-revoke-sub-5035d3f3-pqkd9\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:21.805530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:21.805431    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:22.094917 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:22.094884    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4962e0_e4d0_45de_8909_4b28c3777159.slice/crio-8712de59706f016131e854f49cbb1cab9b51be68a43dea42e3d12a52ad3cbc39 WatchSource:0}: Error finding container 8712de59706f016131e854f49cbb1cab9b51be68a43dea42e3d12a52ad3cbc39: Status 404 returned error can't find the container with id 8712de59706f016131e854f49cbb1cab9b51be68a43dea42e3d12a52ad3cbc39
Oct 31 19:34:22.095937 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:22.095913    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"]
Oct 31 19:34:22.233308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:22.233273    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9" event={"ID":"ad4962e0-e4d0-45de-8909-4b28c3777159","Type":"ContainerStarted","Data":"8712de59706f016131e854f49cbb1cab9b51be68a43dea42e3d12a52ad3cbc39"}
Oct 31 19:34:23.278137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:23.277497    5958 generic.go:356] "Generic (PLEG): container finished" podID="ad4962e0-e4d0-45de-8909-4b28c3777159" containerID="2058747992dc900a47d29972b6bf314438634b1a8478af16e18f003bd0790eb4" exitCode=0
Oct 31 19:34:23.278137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:23.277657    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9" event={"ID":"ad4962e0-e4d0-45de-8909-4b28c3777159","Type":"ContainerDied","Data":"2058747992dc900a47d29972b6bf314438634b1a8478af16e18f003bd0790eb4"}
Oct 31 19:34:24.591038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.591013    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:24.788587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.788169    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca\") pod \"ad4962e0-e4d0-45de-8909-4b28c3777159\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") "
Oct 31 19:34:24.788587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.788299    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca\") pod \"ad4962e0-e4d0-45de-8909-4b28c3777159\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") "
Oct 31 19:34:24.788587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.788518    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s\" (UniqueName: \"kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s\") pod \"ad4962e0-e4d0-45de-8909-4b28c3777159\" (UID: \"ad4962e0-e4d0-45de-8909-4b28c3777159\") "
Oct 31 19:34:24.809301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.808359    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca" pod "ad4962e0-e4d0-45de-8909-4b28c3777159" (UID: "ad4962e0-e4d0-45de-8909-4b28c3777159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:34:24.825790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.823375    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca" pod "ad4962e0-e4d0-45de-8909-4b28c3777159" (UID: "ad4962e0-e4d0-45de-8909-4b28c3777159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:34:24.858808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.858741    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s" pod "ad4962e0-e4d0-45de-8909-4b28c3777159" (UID: "ad4962e0-e4d0-45de-8909-4b28c3777159"). InnerVolumeSpecName "kube-api-access-pmh7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:24.907329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.907293    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmh7s\" (UniqueName: \"kubernetes.io/projected/ad4962e0-e4d0-45de-8909-4b28c3777159-kube-api-access-pmh7s\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:24.907329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.907327    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:24.907329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:24.907338    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4962e0-e4d0-45de-8909-4b28c3777159-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:25.294748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:25.294718    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"
Oct 31 19:34:25.294912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:25.294755    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9" event={"ID":"ad4962e0-e4d0-45de-8909-4b28c3777159","Type":"ContainerDied","Data":"8712de59706f016131e854f49cbb1cab9b51be68a43dea42e3d12a52ad3cbc39"}
Oct 31 19:34:25.294912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:25.294813    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8712de59706f016131e854f49cbb1cab9b51be68a43dea42e3d12a52ad3cbc39"
Oct 31 19:34:27.147607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.147571    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"]
Oct 31 19:34:27.148590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.148561    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ad4962e0-e4d0-45de-8909-4b28c3777159" containerName="revoke-keys"
Oct 31 19:34:27.148590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.148588    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4962e0-e4d0-45de-8909-4b28c3777159" containerName="revoke-keys"
Oct 31 19:34:27.148772 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.148761    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4962e0-e4d0-45de-8909-4b28c3777159" containerName="revoke-keys"
Oct 31 19:34:27.152690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.152664    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.160660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.160631    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"]
Oct 31 19:34:27.165814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.165782    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-85be723d\""
Oct 31 19:34:27.165814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.165785    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-85be723d-dockercfg-9shw9\""
Oct 31 19:34:27.168990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.168967    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"]
Oct 31 19:34:27.173907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.173887    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.185767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.185739    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"]
Oct 31 19:34:27.394627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.391576    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4ts\" (UniqueName: \"kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts\") pod \"payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.394627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.391697    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume\") pod \"payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.394627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.391846    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.394627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.392064    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c97mj\" (UniqueName: \"kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj\") pod \"payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.496570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.495666    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4ts\" (UniqueName: \"kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts\") pod \"payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.496570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.495844    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume\") pod \"payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.496570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.495986    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.496570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.496166    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c97mj\" (UniqueName: \"kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj\") pod \"payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.498666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.498640    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume\") pod \"payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.502694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.499594    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.527488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.527446    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c97mj\" (UniqueName: \"kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj\") pod \"payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") " pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:27.533826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.533793    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4ts\" (UniqueName: \"kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts\") pod \"payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.771895 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.771812    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:27.801087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:27.801059    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:28.168992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.168521    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"]
Oct 31 19:34:28.177526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.177158    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"]
Oct 31 19:34:28.418398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.418313    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"]
Oct 31 19:34:28.424061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.424028    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.428884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.428683    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:34:28.430596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.430535    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:34:28.430986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.430965    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"]
Oct 31 19:34:28.480810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.480776    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf" event={"ID":"e929e1ce-5f72-448d-82ce-9cb8fdf8c259","Type":"ContainerStarted","Data":"332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3"}
Oct 31 19:34:28.481019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.480838    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf" event={"ID":"e929e1ce-5f72-448d-82ce-9cb8fdf8c259","Type":"ContainerStarted","Data":"39a0ebc0ae1ff830a36e8d032a07bdb4cc4ad7da3b4bbdd0a9c61bfde726ffac"}
Oct 31 19:34:28.481019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.480872    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:28.482907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.482878    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh" event={"ID":"3f187783-377f-4164-a553-a3ffd18cf99a","Type":"ContainerStarted","Data":"3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137"}
Oct 31 19:34:28.483065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.482924    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh" event={"ID":"3f187783-377f-4164-a553-a3ffd18cf99a","Type":"ContainerStarted","Data":"a6364ccaa053b2e7987d3d963c21e710c171125a78c4f11aad459a09ebaaff43"}
Oct 31 19:34:28.483065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.483038    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:28.525207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.525151    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh" podStartSLOduration=1.52513749 podStartE2EDuration="1.52513749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:28.523178434 +0000 UTC m=+861.984553050" watchObservedRunningTime="2026-10-31 19:34:28.52513749 +0000 UTC m=+861.986512105"
Oct 31 19:34:28.525805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.525761    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf" podStartSLOduration=1.525750052 podStartE2EDuration="1.525750052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:28.502214401 +0000 UTC m=+861.963589013" watchObservedRunningTime="2026-10-31 19:34:28.525750052 +0000 UTC m=+861.987124670"
Oct 31 19:34:28.574238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.574204    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.574395 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.574245    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9tt4\" (UniqueName: \"kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.574471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.574452    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.676678 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.676530    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.676910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.676887    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9tt4\" (UniqueName: \"kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.677218 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.677202    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.679918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.679891    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.707635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.702838    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.707635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.706513    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9tt4\" (UniqueName: \"kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4\") pod \"maas-api-revoke-sub-dc56b863-m22km\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:28.750567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:28.746948    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:29.118322 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:29.118287    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d51830_2bad_4364_b14b_ed7bf57c9a31.slice/crio-099778d42a16c19532479d3f3ffb6547e209b8f69bf65bf7507027bb71cca2b1 WatchSource:0}: Error finding container 099778d42a16c19532479d3f3ffb6547e209b8f69bf65bf7507027bb71cca2b1: Status 404 returned error can't find the container with id 099778d42a16c19532479d3f3ffb6547e209b8f69bf65bf7507027bb71cca2b1
Oct 31 19:34:29.125736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:29.123717    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"]
Oct 31 19:34:29.501618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:29.501529    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km" event={"ID":"e5d51830-2bad-4364-b14b-ed7bf57c9a31","Type":"ContainerStarted","Data":"099778d42a16c19532479d3f3ffb6547e209b8f69bf65bf7507027bb71cca2b1"}
Oct 31 19:34:30.548665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:30.540795    5958 generic.go:356] "Generic (PLEG): container finished" podID="e5d51830-2bad-4364-b14b-ed7bf57c9a31" containerID="2f9de7c83c96c0e722ee0b869895c19204ac58f99940450b0941b384edb3223e" exitCode=0
Oct 31 19:34:30.548665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:30.541020    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km" event={"ID":"e5d51830-2bad-4364-b14b-ed7bf57c9a31","Type":"ContainerDied","Data":"2f9de7c83c96c0e722ee0b869895c19204ac58f99940450b0941b384edb3223e"}
Oct 31 19:34:31.548495 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.548461    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"]
Oct 31 19:34:31.554458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.554239    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"]
Oct 31 19:34:31.555166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.555125    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.559137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.559114    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.563886 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.563831    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f0303f0d\""
Oct 31 19:34:31.567675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.567649    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"]
Oct 31 19:34:31.575643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.575614    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"]
Oct 31 19:34:31.657319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.655195    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzvj\" (UniqueName: \"kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj\") pod \"payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") " pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.657319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.655424    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.657319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.655825    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") " pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.657319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.655974    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4j5\" (UniqueName: \"kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5\") pod \"payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.801813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.794006    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzvj\" (UniqueName: \"kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj\") pod \"payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") " pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.801813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.794300    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.801813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.794371    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") " pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.801813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.794538    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4j5\" (UniqueName: \"kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5\") pod \"payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.828578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.817004    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") " pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.828578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.817856    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.901043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.898611    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzvj\" (UniqueName: \"kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj\") pod \"payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") " pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:31.917626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.913697    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4j5\" (UniqueName: \"kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5\") pod \"payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.932865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.927100    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:31.932865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:31.927642    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:32.081311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.080932    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:32.155622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.155583    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4\" (UniqueName: \"kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4\") pod \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") "
Oct 31 19:34:32.155804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.155772    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca\") pod \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") "
Oct 31 19:34:32.155895 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.155819    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca\") pod \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\" (UID: \"e5d51830-2bad-4364-b14b-ed7bf57c9a31\") "
Oct 31 19:34:32.156442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.156153    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca" pod "e5d51830-2bad-4364-b14b-ed7bf57c9a31" (UID: "e5d51830-2bad-4364-b14b-ed7bf57c9a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:34:32.156947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.156922    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:32.158954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.158928    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4" pod "e5d51830-2bad-4364-b14b-ed7bf57c9a31" (UID: "e5d51830-2bad-4364-b14b-ed7bf57c9a31"). InnerVolumeSpecName "kube-api-access-p9tt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:34:32.159087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.159072    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca" pod "e5d51830-2bad-4364-b14b-ed7bf57c9a31" (UID: "e5d51830-2bad-4364-b14b-ed7bf57c9a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:34:32.259404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.259365    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9tt4\" (UniqueName: \"kubernetes.io/projected/e5d51830-2bad-4364-b14b-ed7bf57c9a31-kube-api-access-p9tt4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:32.259404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.259400    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5d51830-2bad-4364-b14b-ed7bf57c9a31-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:34:32.490756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.489784    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"]
Oct 31 19:34:32.511002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.505786    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"]
Oct 31 19:34:32.563829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.563792    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z" event={"ID":"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e","Type":"ContainerStarted","Data":"c8ef543cdccb208dd7c9776460d0644d90c803ef982cee01c3dcf0d5311503ea"}
Oct 31 19:34:32.565861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.565835    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km" event={"ID":"e5d51830-2bad-4364-b14b-ed7bf57c9a31","Type":"ContainerDied","Data":"099778d42a16c19532479d3f3ffb6547e209b8f69bf65bf7507027bb71cca2b1"}
Oct 31 19:34:32.566004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.565885    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="099778d42a16c19532479d3f3ffb6547e209b8f69bf65bf7507027bb71cca2b1"
Oct 31 19:34:32.566004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.565904    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"
Oct 31 19:34:32.568999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:32.568966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb" event={"ID":"130ec430-ed77-413f-bc0b-973ec16ea54c","Type":"ContainerStarted","Data":"084a7da8ae4185b6349259db4ce6dfda0454e99717caefa1bbef982ac3838a26"}
Oct 31 19:34:33.578780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:33.578728    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z" event={"ID":"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e","Type":"ContainerStarted","Data":"37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5"}
Oct 31 19:34:33.579246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:33.578959    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:33.580486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:33.580460    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb" event={"ID":"130ec430-ed77-413f-bc0b-973ec16ea54c","Type":"ContainerStarted","Data":"3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9"}
Oct 31 19:34:33.580628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:33.580570    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:33.609529 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:33.609465    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z" podStartSLOduration=2.609452308 podStartE2EDuration="2.609452308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:33.609099749 +0000 UTC m=+867.070474379" watchObservedRunningTime="2026-10-31 19:34:33.609452308 +0000 UTC m=+867.070826930"
Oct 31 19:34:33.649301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:33.649237    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb" podStartSLOduration=2.649223681 podStartE2EDuration="2.649223681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:33.647324885 +0000 UTC m=+867.108699501" watchObservedRunningTime="2026-10-31 19:34:33.649223681 +0000 UTC m=+867.110598335"
Oct 31 19:34:35.711735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.711625    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"]
Oct 31 19:34:35.712489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.712472    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e5d51830-2bad-4364-b14b-ed7bf57c9a31" containerName="revoke-keys"
Oct 31 19:34:35.712587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.712492    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d51830-2bad-4364-b14b-ed7bf57c9a31" containerName="revoke-keys"
Oct 31 19:34:35.712682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.712671    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d51830-2bad-4364-b14b-ed7bf57c9a31" containerName="revoke-keys"
Oct 31 19:34:35.717126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.717087    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.734913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.734859    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-85be723d-serving-cert\""
Oct 31 19:34:35.742486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.742459    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"]
Oct 31 19:34:35.850737 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.850707    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"]
Oct 31 19:34:35.855678 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.855654    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:35.864365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.864336    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9mn\" (UniqueName: \"kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.864521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.864412    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.864639 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.864620    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.864726 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.864712    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.872358 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.872333    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"]
Oct 31 19:34:35.903484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.903197    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1cc6560c\""
Oct 31 19:34:35.923473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.923442    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1cc6560c-dockercfg-pz86b\""
Oct 31 19:34:35.966516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.966336    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqpt\" (UniqueName: \"kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt\") pod \"payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") " pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:35.967002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.966970    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume\") pod \"payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") " pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:35.967326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.967302    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.967616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.967595    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.967918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.967896    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9mn\" (UniqueName: \"kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.968120 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.968083    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.969282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.969261    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:35.978938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:35.978910    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:36.100575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.037215    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:36.109570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.103984    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqpt\" (UniqueName: \"kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt\") pod \"payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") " pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:36.109570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.104079    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume\") pod \"payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") " pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:36.109570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.105426    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume\") pod \"payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") " pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:36.110733 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.110702    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9mn\" (UniqueName: \"kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn\") pod \"maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:36.168568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.154816    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqpt\" (UniqueName: \"kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt\") pod \"payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") " pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:36.192905 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.189660    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:36.349985 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.349944    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:36.899169 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:36.899122    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad60fd70_ea75_4b07_b330_b58ade511e54.slice/crio-9e76d11120c602cc30b335c5d6169e4affcab9b52dea64e34aee408b56d4584b WatchSource:0}: Error finding container 9e76d11120c602cc30b335c5d6169e4affcab9b52dea64e34aee408b56d4584b: Status 404 returned error can't find the container with id 9e76d11120c602cc30b335c5d6169e4affcab9b52dea64e34aee408b56d4584b
Oct 31 19:34:36.900099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:36.900075    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"]
Oct 31 19:34:37.070112 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:34:37.070081    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0e9fd2_d31c_49d4_8b8f_af90f3f283e4.slice/crio-3af7e48f5965f769fe4de346e01d6a3073d1b722ad945598bc47a6c0588690f7 WatchSource:0}: Error finding container 3af7e48f5965f769fe4de346e01d6a3073d1b722ad945598bc47a6c0588690f7: Status 404 returned error can't find the container with id 3af7e48f5965f769fe4de346e01d6a3073d1b722ad945598bc47a6c0588690f7
Oct 31 19:34:37.072278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.072252    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:34:37.072538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.072516    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"]
Oct 31 19:34:37.780274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.780239    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55" event={"ID":"ad60fd70-ea75-4b07-b330-b58ade511e54","Type":"ContainerStarted","Data":"631335a338aab374c077c91ebe41a3d5eec45369a9823380acc082703cd6ad52"}
Oct 31 19:34:37.780474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.780300    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55" event={"ID":"ad60fd70-ea75-4b07-b330-b58ade511e54","Type":"ContainerStarted","Data":"9e76d11120c602cc30b335c5d6169e4affcab9b52dea64e34aee408b56d4584b"}
Oct 31 19:34:37.780474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.780331    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:37.782352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.782327    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2" event={"ID":"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4","Type":"ContainerStarted","Data":"359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f"}
Oct 31 19:34:37.782501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.782375    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2" event={"ID":"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4","Type":"ContainerStarted","Data":"3af7e48f5965f769fe4de346e01d6a3073d1b722ad945598bc47a6c0588690f7"}
Oct 31 19:34:37.782501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.782405    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:37.810152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.810041    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55" podStartSLOduration=2.8100266229999997 podStartE2EDuration="2.810026623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:37.802336741 +0000 UTC m=+871.263711357" watchObservedRunningTime="2026-10-31 19:34:37.810026623 +0000 UTC m=+871.271401247"
Oct 31 19:34:37.831059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:37.830993    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2" podStartSLOduration=2.360792814 podStartE2EDuration="2.830972595s" totalImagesPullingTime="470.179781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:37.827845169 +0000 UTC m=+871.289219795" watchObservedRunningTime="2026-10-31 19:34:37.830972595 +0000 UTC m=+871.292347211"
Oct 31 19:34:39.505284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.505253    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:34:39.505732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.505333    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:34:39.801503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.801426    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"]
Oct 31 19:34:39.806108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.806088    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.811370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.811346    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f0303f0d-serving-cert\""
Oct 31 19:34:39.818050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.818022    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"]
Oct 31 19:34:39.861915 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.861879    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zs4\" (UniqueName: \"kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.862106 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.862086    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.862285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.862263    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.862351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.862340    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.963899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.963864    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zs4\" (UniqueName: \"kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.964075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.963984    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.964141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.964072    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.964141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.964119    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.964778 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.964742    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.966730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.966699    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.966839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.966812    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:39.978008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:39.977978    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zs4\" (UniqueName: \"kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4\") pod \"maas-api-e2e-ait-f0303f0d-d84896ddf-h954v\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:40.125188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:40.125118    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:40.611602 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:40.610811    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"]
Oct 31 19:34:40.808957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:40.808920    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v" event={"ID":"a7044256-dae5-416f-89c3-f4364e0bd30a","Type":"ContainerStarted","Data":"67934fbf19fd0c2989f6072c8b1c5d1a7ab449f5f1d2055dc9d728a851f25a1c"}
Oct 31 19:34:41.831624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:41.820608    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v" event={"ID":"a7044256-dae5-416f-89c3-f4364e0bd30a","Type":"ContainerStarted","Data":"a7e5e4836cf6f4af8bbc0bb64e3b652999fc2b9642fbfd71ce04c4a99160231c"}
Oct 31 19:34:41.832290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:41.832256    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:43.577237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.577183    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v" podStartSLOduration=4.056606331 podStartE2EDuration="4.577167698s" totalImagesPullingTime="520.561367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:41.885004708 +0000 UTC m=+875.346379326" watchObservedRunningTime="2026-10-31 19:34:43.577167698 +0000 UTC m=+877.038542312"
Oct 31 19:34:43.579828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.579804    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"]
Oct 31 19:34:43.586422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.586395    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.596611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.596584    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"]
Oct 31 19:34:43.612144 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.609174    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1cc6560c-serving-cert\""
Oct 31 19:34:43.745564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.745102    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.745564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.745287    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4f2r\" (UniqueName: \"kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.745564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.745400    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.745564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.745461    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.848647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.847819    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:34:43.851037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.851004    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.852173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.851681    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.852173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.851912    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.852173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.852063    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f2r\" (UniqueName: \"kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.870592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.853253    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.870592 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:34:43.853368    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-serving-cert: secret "maas-api-e2e-ait-1cc6560c-serving-cert" not found
Oct 31 19:34:43.870592 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:34:43.853431    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls podName:69d4933c-21c0-4d77-be02-395c15d250dc nodeName:}" failed. No retries permitted until 2026-10-31 19:34:44.353412169 +0000 UTC m=+877.814786762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls") pod "maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" (UID: "69d4933c-21c0-4d77-be02-395c15d250dc") : secret "maas-api-e2e-ait-1cc6560c-serving-cert" not found
Oct 31 19:34:43.904586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.901290    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:43.904586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:43.902021    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f2r\" (UniqueName: \"kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:44.402668 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:44.392893    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:44.563570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:44.559921    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\") pod \"maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:44.623567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:44.614778    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:34:44.671700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:44.627744    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:34:44.948631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:44.938371    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:45.445901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:45.445770    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 31 19:34:45.446574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:45.446520    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.492568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:45.482981    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:34:45.492568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:45.485191    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.757620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:45.752648    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.935924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:45.923640    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.022910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.022812    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.046583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.043711    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.046583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.044122    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:34:46.205866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.205817    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:46.502567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.500141    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.502567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.501146    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:34:46.523250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.523212    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.529463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.529418    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:34:46.798774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.798663    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.989308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.894270    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.029572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:46.929492    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.069874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.030177    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 31 19:34:47.069874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.030233    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.089692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.089627    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.656737 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.656688    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.833226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.822899    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.840117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.840072    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.956570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.840358    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.956570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.840403    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.956835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.859217    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.956835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.952756    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-ggpfz" podUID="c104a69f-80bd-47cc-a6f0-b5e6e199e9c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.972699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:47.972646    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6" podUID="3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.223690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.223582    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:34:48.224109 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.224082    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:48.672819 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:34:48.672778    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:34:48.734980 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:34:48.734937    5958 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.08s"
Oct 31 19:34:48.735306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.735286    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:34:48.735417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.735391    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:34:48.735535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.735521    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:34:48.735770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.735756    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:34:48.749338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.735887    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:34:48.757647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.748576    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:48.872458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.872264    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.956746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.954256    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:34:48.956746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.954320    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"]
Oct 31 19:34:48.956746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.954419    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:34:48.956746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.954467    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:34:48.956746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:48.954513    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:34:49.078775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.078727    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.096643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.094882    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:34:49.103629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.103523    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.104281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.103716    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:34:49.107066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.105737    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:34:49.270759 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.252015    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.270759 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.254317    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.270759 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.254730    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:34:49.373063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.373032    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:34:49.379712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.379677    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:34:49.720683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.720637    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.826605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.808935    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:34:49.896511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:49.896001    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:34:50.057376 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.057343    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:34:50.081207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.081148    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:34:50.082874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.082746    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:50.123566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.112810    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:50.123566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.119176    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 19:34:50.135083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.135043    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" event={"ID":"69d4933c-21c0-4d77-be02-395c15d250dc","Type":"ContainerStarted","Data":"ecc66a6b1b5cf8d57dc65e9149d6316d4ac27d7f8ae31a1f70e8a7afe5e0d519"}
Oct 31 19:34:50.321701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.316994    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:34:50.521427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.376233    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:34:50.553252 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.553224    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:34:50.553536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.553509    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:34:50.557507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.557470    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:34:50.617876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:50.617836    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:34:51.252565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:51.227776    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" event={"ID":"69d4933c-21c0-4d77-be02-395c15d250dc","Type":"ContainerStarted","Data":"519c95d263c6926409ed204550cd1ff3e40a823b80b682e31ff801a4766f8172"}
Oct 31 19:34:51.252565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:51.229170    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:34:51.285605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:51.285061    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" podStartSLOduration=6.934658916 podStartE2EDuration="8.285042746s" totalImagesPullingTime="1.35038383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:51.276036497 +0000 UTC m=+884.737411113" watchObservedRunningTime="2026-10-31 19:34:51.285042746 +0000 UTC m=+884.746417361"
Oct 31 19:34:58.299554 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:34:58.299512    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:35:06.813149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:06.813109    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"]
Oct 31 19:35:06.832408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:06.832373    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"]
Oct 31 19:35:11.829538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:11.818621    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf" podUID="e929e1ce-5f72-448d-82ce-9cb8fdf8c259" containerName="payload-processing" containerID="cri-o://332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3" gracePeriod=25
Oct 31 19:35:11.832914 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:11.832842    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh" podUID="3f187783-377f-4164-a553-a3ffd18cf99a" containerName="payload-pre-processing" containerID="cri-o://3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137" gracePeriod=25
Oct 31 19:35:11.922424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:11.922388    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"]
Oct 31 19:35:11.953048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:11.952399    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"]
Oct 31 19:35:12.219904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.219869    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:35:12.310615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.267516    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:35:12.414894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.318843    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume\") pod \"3f187783-377f-4164-a553-a3ffd18cf99a\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") "
Oct 31 19:35:12.414894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.319107    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts\" (UniqueName: \"kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts\") pod \"3f187783-377f-4164-a553-a3ffd18cf99a\" (UID: \"3f187783-377f-4164-a553-a3ffd18cf99a\") "
Oct 31 19:35:12.414894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.319313    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume\") pod \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") "
Oct 31 19:35:12.414894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.319396    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj\" (UniqueName: \"kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj\") pod \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\" (UID: \"e929e1ce-5f72-448d-82ce-9cb8fdf8c259\") "
Oct 31 19:35:12.414894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.373796    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume" pod "e929e1ce-5f72-448d-82ce-9cb8fdf8c259" (UID: "e929e1ce-5f72-448d-82ce-9cb8fdf8c259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:12.414894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.382831    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume" pod "3f187783-377f-4164-a553-a3ffd18cf99a" (UID: "3f187783-377f-4164-a553-a3ffd18cf99a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:12.462926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.462855    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:12.462926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.462891    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f187783-377f-4164-a553-a3ffd18cf99a-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:12.513610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.483222    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts" pod "3f187783-377f-4164-a553-a3ffd18cf99a" (UID: "3f187783-377f-4164-a553-a3ffd18cf99a"). InnerVolumeSpecName "kube-api-access-mx4ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:12.513610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.483934    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj" pod "e929e1ce-5f72-448d-82ce-9cb8fdf8c259" (UID: "e929e1ce-5f72-448d-82ce-9cb8fdf8c259"). InnerVolumeSpecName "kube-api-access-c97mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:12.544173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.531922    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"]
Oct 31 19:35:12.544173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.532724    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2" podUID="ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" containerName="maas-api" containerID="cri-o://359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f" gracePeriod=30
Oct 31 19:35:12.565283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.565189    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4ts\" (UniqueName: \"kubernetes.io/projected/3f187783-377f-4164-a553-a3ffd18cf99a-kube-api-access-mx4ts\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:12.565283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.565226    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c97mj\" (UniqueName: \"kubernetes.io/projected/e929e1ce-5f72-448d-82ce-9cb8fdf8c259-kube-api-access-c97mj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:12.886599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.882824    5958 generic.go:356] "Generic (PLEG): container finished" podID="3f187783-377f-4164-a553-a3ffd18cf99a" containerID="3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137" exitCode=0
Oct 31 19:35:12.886599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.883109    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"
Oct 31 19:35:12.894611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.894576    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh" event={"ID":"3f187783-377f-4164-a553-a3ffd18cf99a","Type":"ContainerDied","Data":"3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137"}
Oct 31 19:35:12.894860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.894838    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh" event={"ID":"3f187783-377f-4164-a553-a3ffd18cf99a","Type":"ContainerDied","Data":"a6364ccaa053b2e7987d3d963c21e710c171125a78c4f11aad459a09ebaaff43"}
Oct 31 19:35:12.894984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.894971    5958 scope.go:122] "RemoveContainer" containerID="3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137"
Oct 31 19:35:12.903701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.903663    5958 generic.go:356] "Generic (PLEG): container finished" podID="e929e1ce-5f72-448d-82ce-9cb8fdf8c259" containerID="332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3" exitCode=0
Oct 31 19:35:12.904132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.904113    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf" event={"ID":"e929e1ce-5f72-448d-82ce-9cb8fdf8c259","Type":"ContainerDied","Data":"332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3"}
Oct 31 19:35:12.905617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.904356    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf" event={"ID":"e929e1ce-5f72-448d-82ce-9cb8fdf8c259","Type":"ContainerDied","Data":"39a0ebc0ae1ff830a36e8d032a07bdb4cc4ad7da3b4bbdd0a9c61bfde726ffac"}
Oct 31 19:35:12.905617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.904920    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"
Oct 31 19:35:12.939162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.939133    5958 scope.go:122] "RemoveContainer" containerID="3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137"
Oct 31 19:35:12.939664 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:12.939633    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137\": container with ID starting with 3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137 not found: ID does not exist" containerID="3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137"
Oct 31 19:35:12.939813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.939675    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137"} err="failed to get container status \"3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137\": rpc error: code = NotFound desc = could not find container \"3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137\": container with ID starting with 3c2da5276d9b9edc94d1c105da856ce851ad6a116ca6b20e30f1ba4cdda7b137 not found: ID does not exist"
Oct 31 19:35:12.939813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.939699    5958 scope.go:122] "RemoveContainer" containerID="332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3"
Oct 31 19:35:12.970409 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.969970    5958 scope.go:122] "RemoveContainer" containerID="332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3"
Oct 31 19:35:12.970564 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:12.970435    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3\": container with ID starting with 332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3 not found: ID does not exist" containerID="332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3"
Oct 31 19:35:12.970564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.970476    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3"} err="failed to get container status \"332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3\": rpc error: code = NotFound desc = could not find container \"332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3\": container with ID starting with 332b40de747a04cb1af068b8d9164d361509ddee86d88fdf05ed9828bf92f9d3 not found: ID does not exist"
Oct 31 19:35:12.975413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.975386    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"]
Oct 31 19:35:12.984606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.983653    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-85be723d-545664d4ff-7jldh"]
Oct 31 19:35:12.996793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:12.996757    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"]
Oct 31 19:35:13.004369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.004322    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-85be723d-578cfdbb9c-bh7qf"]
Oct 31 19:35:13.085604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.085524    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:35:13.207038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.207004    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls\") pod \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") "
Oct 31 19:35:13.207275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.207142    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn\" (UniqueName: \"kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn\") pod \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") "
Oct 31 19:35:13.207328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.207288    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca\") pod \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") "
Oct 31 19:35:13.207422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.207401    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs\") pod \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\" (UID: \"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4\") "
Oct 31 19:35:13.207727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.207700    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca" pod "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" (UID: "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:13.208152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.208120    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:13.209354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.209330    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls" pod "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" (UID: "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:13.209753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.209733    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs" pod "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" (UID: "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:13.209815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.209765    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn" pod "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" (UID: "ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4"). InnerVolumeSpecName "kube-api-access-nw9mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:13.315596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.314001    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:13.315596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.314038    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:13.315596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.314052    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9mn\" (UniqueName: \"kubernetes.io/projected/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4-kube-api-access-nw9mn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:13.916496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.916463    5958 generic.go:356] "Generic (PLEG): container finished" podID="ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" containerID="359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f" exitCode=0
Oct 31 19:35:13.917043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.916517    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"
Oct 31 19:35:13.917043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.916589    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2" event={"ID":"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4","Type":"ContainerDied","Data":"359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f"}
Oct 31 19:35:13.917043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.916653    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2" event={"ID":"ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4","Type":"ContainerDied","Data":"3af7e48f5965f769fe4de346e01d6a3073d1b722ad945598bc47a6c0588690f7"}
Oct 31 19:35:13.917043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.916683    5958 scope.go:122] "RemoveContainer" containerID="359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f"
Oct 31 19:35:13.929237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.929211    5958 scope.go:122] "RemoveContainer" containerID="359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f"
Oct 31 19:35:13.929536 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:13.929517    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f\": container with ID starting with 359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f not found: ID does not exist" containerID="359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f"
Oct 31 19:35:13.929616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.929562    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f"} err="failed to get container status \"359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f\": rpc error: code = NotFound desc = could not find container \"359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f\": container with ID starting with 359cb978da1fa9c66c0edd184925c41bab54cc76e57d3781ca033f20b6e3a36f not found: ID does not exist"
Oct 31 19:35:13.945241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.945207    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"]
Oct 31 19:35:13.949785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:13.949764    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-85be723d-674d4c4794-4ddp2"]
Oct 31 19:35:14.753607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:14.741645    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f187783-377f-4164-a553-a3ffd18cf99a" path="/var/lib/kubelet/pods/3f187783-377f-4164-a553-a3ffd18cf99a/volumes"
Oct 31 19:35:14.753607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:14.742415    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e929e1ce-5f72-448d-82ce-9cb8fdf8c259" path="/var/lib/kubelet/pods/e929e1ce-5f72-448d-82ce-9cb8fdf8c259/volumes"
Oct 31 19:35:14.753607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:14.745764    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" path="/var/lib/kubelet/pods/ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4/volumes"
Oct 31 19:35:15.738244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:15.738205    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"]
Oct 31 19:35:15.792226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:15.792177    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"]
Oct 31 19:35:15.792478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:15.792445    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v" podUID="a7044256-dae5-416f-89c3-f4364e0bd30a" containerName="maas-api" containerID="cri-o://a7e5e4836cf6f4af8bbc0bb64e3b652999fc2b9642fbfd71ce04c4a99160231c" gracePeriod=30
Oct 31 19:35:15.979867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:15.979834    5958 generic.go:356] "Generic (PLEG): container finished" podID="a7044256-dae5-416f-89c3-f4364e0bd30a" containerID="a7e5e4836cf6f4af8bbc0bb64e3b652999fc2b9642fbfd71ce04c4a99160231c" exitCode=0
Oct 31 19:35:15.980037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:15.979886    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v" event={"ID":"a7044256-dae5-416f-89c3-f4364e0bd30a","Type":"ContainerDied","Data":"a7e5e4836cf6f4af8bbc0bb64e3b652999fc2b9642fbfd71ce04c4a99160231c"}
Oct 31 19:35:16.505031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.503771    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:35:16.636586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.633254    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca\") pod \"a7044256-dae5-416f-89c3-f4364e0bd30a\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") "
Oct 31 19:35:16.636586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.633361    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls\") pod \"a7044256-dae5-416f-89c3-f4364e0bd30a\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") "
Oct 31 19:35:16.636586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.633565    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs\") pod \"a7044256-dae5-416f-89c3-f4364e0bd30a\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") "
Oct 31 19:35:16.636586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.633616    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4\" (UniqueName: \"kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4\") pod \"a7044256-dae5-416f-89c3-f4364e0bd30a\" (UID: \"a7044256-dae5-416f-89c3-f4364e0bd30a\") "
Oct 31 19:35:16.648574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.640213    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca" pod "a7044256-dae5-416f-89c3-f4364e0bd30a" (UID: "a7044256-dae5-416f-89c3-f4364e0bd30a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:16.737965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.737901    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7044256-dae5-416f-89c3-f4364e0bd30a-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:16.752138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.752100    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs" pod "a7044256-dae5-416f-89c3-f4364e0bd30a" (UID: "a7044256-dae5-416f-89c3-f4364e0bd30a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:16.828511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.827783    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4" pod "a7044256-dae5-416f-89c3-f4364e0bd30a" (UID: "a7044256-dae5-416f-89c3-f4364e0bd30a"). InnerVolumeSpecName "kube-api-access-h4zs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:16.849957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.832005    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls" pod "a7044256-dae5-416f-89c3-f4364e0bd30a" (UID: "a7044256-dae5-416f-89c3-f4364e0bd30a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:16.869565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.857015    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:16.869565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.857055    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zs4\" (UniqueName: \"kubernetes.io/projected/a7044256-dae5-416f-89c3-f4364e0bd30a-kube-api-access-h4zs4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:16.869565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.857072    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7044256-dae5-416f-89c3-f4364e0bd30a-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:16.952320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.947198    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z" podUID="8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" containerName="payload-processing" containerID="cri-o://37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5" gracePeriod=25
Oct 31 19:35:16.955272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.954125    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb" podUID="130ec430-ed77-413f-bc0b-973ec16ea54c" containerName="payload-pre-processing" containerID="cri-o://3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9" gracePeriod=25
Oct 31 19:35:16.992439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.992410    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"
Oct 31 19:35:16.992631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.992443    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v" event={"ID":"a7044256-dae5-416f-89c3-f4364e0bd30a","Type":"ContainerDied","Data":"67934fbf19fd0c2989f6072c8b1c5d1a7ab449f5f1d2055dc9d728a851f25a1c"}
Oct 31 19:35:16.992631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:16.992516    5958 scope.go:122] "RemoveContainer" containerID="a7e5e4836cf6f4af8bbc0bb64e3b652999fc2b9642fbfd71ce04c4a99160231c"
Oct 31 19:35:17.066726 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:17.066676    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:35:17.120623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.120584    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"]
Oct 31 19:35:17.126238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.126207    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0303f0d-d84896ddf-h954v"]
Oct 31 19:35:17.585978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.584972    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:35:17.646096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.646068    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:35:17.754033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.753997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume\") pod \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") "
Oct 31 19:35:17.754479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.754265    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj\" (UniqueName: \"kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj\") pod \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\" (UID: \"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e\") "
Oct 31 19:35:17.754479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.754328    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume\") pod \"130ec430-ed77-413f-bc0b-973ec16ea54c\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") "
Oct 31 19:35:17.754479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.754396    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5\" (UniqueName: \"kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5\") pod \"130ec430-ed77-413f-bc0b-973ec16ea54c\" (UID: \"130ec430-ed77-413f-bc0b-973ec16ea54c\") "
Oct 31 19:35:17.754479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.754396    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume" pod "8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" (UID: "8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:17.754829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.754795    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume" pod "130ec430-ed77-413f-bc0b-973ec16ea54c" (UID: "130ec430-ed77-413f-bc0b-973ec16ea54c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:17.755180 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.755159    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:17.755180 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.755179    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130ec430-ed77-413f-bc0b-973ec16ea54c-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:17.756659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.756636    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5" pod "130ec430-ed77-413f-bc0b-973ec16ea54c" (UID: "130ec430-ed77-413f-bc0b-973ec16ea54c"). InnerVolumeSpecName "kube-api-access-st4j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:17.756752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.756694    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj" pod "8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" (UID: "8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e"). InnerVolumeSpecName "kube-api-access-4xzvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:17.856679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.856636    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xzvj\" (UniqueName: \"kubernetes.io/projected/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e-kube-api-access-4xzvj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:17.856679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:17.856680    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4j5\" (UniqueName: \"kubernetes.io/projected/130ec430-ed77-413f-bc0b-973ec16ea54c-kube-api-access-st4j5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.002875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.002836    5958 generic.go:356] "Generic (PLEG): container finished" podID="8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" containerID="37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5" exitCode=0
Oct 31 19:35:18.003061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.002957    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"
Oct 31 19:35:18.003061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.002966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z" event={"ID":"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e","Type":"ContainerDied","Data":"37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5"}
Oct 31 19:35:18.003061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.003042    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z" event={"ID":"8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e","Type":"ContainerDied","Data":"c8ef543cdccb208dd7c9776460d0644d90c803ef982cee01c3dcf0d5311503ea"}
Oct 31 19:35:18.003215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.003065    5958 scope.go:122] "RemoveContainer" containerID="37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5"
Oct 31 19:35:18.006345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.006315    5958 generic.go:356] "Generic (PLEG): container finished" podID="130ec430-ed77-413f-bc0b-973ec16ea54c" containerID="3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9" exitCode=0
Oct 31 19:35:18.006461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.006369    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb" event={"ID":"130ec430-ed77-413f-bc0b-973ec16ea54c","Type":"ContainerDied","Data":"3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9"}
Oct 31 19:35:18.006461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.006394    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"
Oct 31 19:35:18.006461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.006416    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb" event={"ID":"130ec430-ed77-413f-bc0b-973ec16ea54c","Type":"ContainerDied","Data":"084a7da8ae4185b6349259db4ce6dfda0454e99717caefa1bbef982ac3838a26"}
Oct 31 19:35:18.018693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.018669    5958 scope.go:122] "RemoveContainer" containerID="37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5"
Oct 31 19:35:18.019361 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:18.019339    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5\": container with ID starting with 37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5 not found: ID does not exist" containerID="37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5"
Oct 31 19:35:18.019442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.019370    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5"} err="failed to get container status \"37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5\": rpc error: code = NotFound desc = could not find container \"37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5\": container with ID starting with 37fd253faf0b4d429db27c719c7e0d3c371deaa97517b29c707695b7d8678bc5 not found: ID does not exist"
Oct 31 19:35:18.019442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.019387    5958 scope.go:122] "RemoveContainer" containerID="3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9"
Oct 31 19:35:18.032845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.032819    5958 scope.go:122] "RemoveContainer" containerID="3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9"
Oct 31 19:35:18.033230 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:18.033210    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9\": container with ID starting with 3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9 not found: ID does not exist" containerID="3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9"
Oct 31 19:35:18.033292 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.033241    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9"} err="failed to get container status \"3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9\": rpc error: code = NotFound desc = could not find container \"3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9\": container with ID starting with 3f13c23ba2b394acd7e194ebf689b1f8e9471b4a86355315c429c75dac40c8b9 not found: ID does not exist"
Oct 31 19:35:18.039974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.039946    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"]
Oct 31 19:35:18.048484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.048414    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0303f0d-6f65d9f4c-z7c4z"]
Oct 31 19:35:18.069293 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.069254    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"]
Oct 31 19:35:18.077040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.076967    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0303f0d-7f59c48747-wntjb"]
Oct 31 19:35:18.805895 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.805864    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130ec430-ed77-413f-bc0b-973ec16ea54c" path="/var/lib/kubelet/pods/130ec430-ed77-413f-bc0b-973ec16ea54c/volumes"
Oct 31 19:35:18.807143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.807041    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" path="/var/lib/kubelet/pods/8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e/volumes"
Oct 31 19:35:18.807627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:18.807609    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7044256-dae5-416f-89c3-f4364e0bd30a" path="/var/lib/kubelet/pods/a7044256-dae5-416f-89c3-f4364e0bd30a/volumes"
Oct 31 19:35:19.452304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.452264    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"]
Oct 31 19:35:19.453063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453047    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e929e1ce-5f72-448d-82ce-9cb8fdf8c259" containerName="payload-processing"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453064    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e929e1ce-5f72-448d-82ce-9cb8fdf8c259" containerName="payload-processing"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453079    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3f187783-377f-4164-a553-a3ffd18cf99a" containerName="payload-pre-processing"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453084    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f187783-377f-4164-a553-a3ffd18cf99a" containerName="payload-pre-processing"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453096    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" containerName="payload-processing"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453101    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" containerName="payload-processing"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453110    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" containerName="maas-api"
Oct 31 19:35:19.453114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453115    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" containerName="maas-api"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453123    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a7044256-dae5-416f-89c3-f4364e0bd30a" containerName="maas-api"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453128    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7044256-dae5-416f-89c3-f4364e0bd30a" containerName="maas-api"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453146    5958 cpu_manager.go:414] "containerMap: removing container" podUID="130ec430-ed77-413f-bc0b-973ec16ea54c" containerName="payload-pre-processing"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453151    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130ec430-ed77-413f-bc0b-973ec16ea54c" containerName="payload-pre-processing"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453257    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e929e1ce-5f72-448d-82ce-9cb8fdf8c259" containerName="payload-processing"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453265    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f187783-377f-4164-a553-a3ffd18cf99a" containerName="payload-pre-processing"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453271    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be6efd4-2dbd-4ab4-8a65-46b898dc0e4e" containerName="payload-processing"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453277    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0e9fd2-d31c-49d4-8b8f-af90f3f283e4" containerName="maas-api"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453283    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7044256-dae5-416f-89c3-f4364e0bd30a" containerName="maas-api"
Oct 31 19:35:19.453320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.453294    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="130ec430-ed77-413f-bc0b-973ec16ea54c" containerName="payload-pre-processing"
Oct 31 19:35:19.469339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.469298    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.481705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.477464    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"]
Oct 31 19:35:19.481705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.477667    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-63eab93e\""
Oct 31 19:35:19.481705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.479719    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-63eab93e-dockercfg-h5cwn\""
Oct 31 19:35:19.540194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.540164    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:35:19.545705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.545671    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"]
Oct 31 19:35:19.550710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.550685    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:19.562197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.562158    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"]
Oct 31 19:35:19.632093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.632045    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.634880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.634847    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwx9d\" (UniqueName: \"kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d\") pod \"payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.743618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.738968    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume\") pod \"payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") " pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:19.743618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.739110    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmhxz\" (UniqueName: \"kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz\") pod \"payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") " pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:19.743618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.739154    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.743618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.739250    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwx9d\" (UniqueName: \"kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d\") pod \"payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.757571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.756932    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.792565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.786597    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwx9d\" (UniqueName: \"kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d\") pod \"payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.798876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.798839    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:19.894797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.890843    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume\") pod \"payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") " pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:19.894797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.891014    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmhxz\" (UniqueName: \"kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz\") pod \"payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") " pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:19.894797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.892387    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume\") pod \"payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") " pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:19.956590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:19.949848    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmhxz\" (UniqueName: \"kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz\") pod \"payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") " pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:20.174889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:20.174849    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:20.576569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:20.575623    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"]
Oct 31 19:35:20.752929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:20.752788    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55" podUID="ad60fd70-ea75-4b07-b330-b58ade511e54" containerName="payload-processing" containerID="cri-o://631335a338aab374c077c91ebe41a3d5eec45369a9823380acc082703cd6ad52" gracePeriod=25
Oct 31 19:35:21.094801 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.080972    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"]
Oct 31 19:35:21.223234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.223195    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"]
Oct 31 19:35:21.224042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.223669    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" podUID="69d4933c-21c0-4d77-be02-395c15d250dc" containerName="maas-api" containerID="cri-o://519c95d263c6926409ed204550cd1ff3e40a823b80b682e31ff801a4766f8172" gracePeriod=30
Oct 31 19:35:21.310796 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.310674    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw" event={"ID":"21d105c4-79ff-4690-b3d3-6fe001aa7ef2","Type":"ContainerStarted","Data":"e0bc1b66e6147856d3114ed4a08879a68eeba21d0fa691cc26f6d010e0c67fb6"}
Oct 31 19:35:21.322413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.322052    5958 generic.go:356] "Generic (PLEG): container finished" podID="ad60fd70-ea75-4b07-b330-b58ade511e54" containerID="631335a338aab374c077c91ebe41a3d5eec45369a9823380acc082703cd6ad52" exitCode=0
Oct 31 19:35:21.322413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.322325    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55" event={"ID":"ad60fd70-ea75-4b07-b330-b58ade511e54","Type":"ContainerDied","Data":"631335a338aab374c077c91ebe41a3d5eec45369a9823380acc082703cd6ad52"}
Oct 31 19:35:21.330045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:21.326672    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p" event={"ID":"e0345914-4dac-4174-915d-d16fcd9decd0","Type":"ContainerStarted","Data":"669006e05d5a95b04f74e93dd0874ab218dd079e7c6de313c298a1ef7ce314e8"}
Oct 31 19:35:22.298632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.289794    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:35:22.655535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.655476    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume\") pod \"ad60fd70-ea75-4b07-b330-b58ade511e54\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") "
Oct 31 19:35:22.657072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.656662    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt\" (UniqueName: \"kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt\") pod \"ad60fd70-ea75-4b07-b330-b58ade511e54\" (UID: \"ad60fd70-ea75-4b07-b330-b58ade511e54\") "
Oct 31 19:35:22.659199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.659168    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume" pod "ad60fd70-ea75-4b07-b330-b58ade511e54" (UID: "ad60fd70-ea75-4b07-b330-b58ade511e54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:22.661291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.661261    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad60fd70-ea75-4b07-b330-b58ade511e54-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:22.796028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.795988    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt" pod "ad60fd70-ea75-4b07-b330-b58ade511e54" (UID: "ad60fd70-ea75-4b07-b330-b58ade511e54"). InnerVolumeSpecName "kube-api-access-4fqpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:22.971699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:22.971016    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.021535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.020764    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p" event={"ID":"e0345914-4dac-4174-915d-d16fcd9decd0","Type":"ContainerStarted","Data":"392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc"}
Oct 31 19:35:23.021535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.020848    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:23.033452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.023938    5958 generic.go:356] "Generic (PLEG): container finished" podID="69d4933c-21c0-4d77-be02-395c15d250dc" containerID="519c95d263c6926409ed204550cd1ff3e40a823b80b682e31ff801a4766f8172" exitCode=0
Oct 31 19:35:23.033452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.024182    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" event={"ID":"69d4933c-21c0-4d77-be02-395c15d250dc","Type":"ContainerDied","Data":"519c95d263c6926409ed204550cd1ff3e40a823b80b682e31ff801a4766f8172"}
Oct 31 19:35:23.043867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.043832    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fqpt\" (UniqueName: \"kubernetes.io/projected/ad60fd70-ea75-4b07-b330-b58ade511e54-kube-api-access-4fqpt\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:23.090589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.056067    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw" event={"ID":"21d105c4-79ff-4690-b3d3-6fe001aa7ef2","Type":"ContainerStarted","Data":"7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f"}
Oct 31 19:35:23.090589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.061565    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:23.090589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.069741    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:35:23.108739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.108685    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55" event={"ID":"ad60fd70-ea75-4b07-b330-b58ade511e54","Type":"ContainerDied","Data":"9e76d11120c602cc30b335c5d6169e4affcab9b52dea64e34aee408b56d4584b"}
Oct 31 19:35:23.109033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.109011    5958 scope.go:122] "RemoveContainer" containerID="631335a338aab374c077c91ebe41a3d5eec45369a9823380acc082703cd6ad52"
Oct 31 19:35:23.109491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.109471    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"
Oct 31 19:35:23.189570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.178855    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\") pod \"69d4933c-21c0-4d77-be02-395c15d250dc\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") "
Oct 31 19:35:23.189570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.178945    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca\") pod \"69d4933c-21c0-4d77-be02-395c15d250dc\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") "
Oct 31 19:35:23.189570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.179063    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs\") pod \"69d4933c-21c0-4d77-be02-395c15d250dc\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") "
Oct 31 19:35:23.262587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.250531    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca" pod "69d4933c-21c0-4d77-be02-395c15d250dc" (UID: "69d4933c-21c0-4d77-be02-395c15d250dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:23.275659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.274074    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.328770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.328735    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r\" (UniqueName: \"kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r\") pod \"69d4933c-21c0-4d77-be02-395c15d250dc\" (UID: \"69d4933c-21c0-4d77-be02-395c15d250dc\") "
Oct 31 19:35:23.384028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.379011    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d4933c-21c0-4d77-be02-395c15d250dc-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:23.396788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.396733    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls" pod "69d4933c-21c0-4d77-be02-395c15d250dc" (UID: "69d4933c-21c0-4d77-be02-395c15d250dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:23.506898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.487826    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r" pod "69d4933c-21c0-4d77-be02-395c15d250dc" (UID: "69d4933c-21c0-4d77-be02-395c15d250dc"). InnerVolumeSpecName "kube-api-access-p4f2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:23.506898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.496170    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p" podStartSLOduration=4.496142929 podStartE2EDuration="4.496142929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:23.15479675 +0000 UTC m=+916.616171366" watchObservedRunningTime="2026-10-31 19:35:23.496142929 +0000 UTC m=+916.957517548"
Oct 31 19:35:23.506898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.497297    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw" podStartSLOduration=4.497286621 podStartE2EDuration="4.497286621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:23.358160994 +0000 UTC m=+916.819535609" watchObservedRunningTime="2026-10-31 19:35:23.497286621 +0000 UTC m=+916.958661237"
Oct 31 19:35:23.548773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.548687    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.550035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.549990    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs" pod "69d4933c-21c0-4d77-be02-395c15d250dc" (UID: "69d4933c-21c0-4d77-be02-395c15d250dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:35:23.576338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.576255    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.583839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.583788    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:35:23.602581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.593183    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:23.675577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.639690    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d4933c-21c0-4d77-be02-395c15d250dc-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:23.675577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:23.639738    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4f2r\" (UniqueName: \"kubernetes.io/projected/69d4933c-21c0-4d77-be02-395c15d250dc-kube-api-access-p4f2r\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:24.095574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.030491    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.190142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.163162    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"]
Oct 31 19:35:24.285890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.285847    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1cc6560c-7f7db84869-wln55"]
Oct 31 19:35:24.381577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.380178    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.381577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.381329    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"
Oct 31 19:35:24.411577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.410746    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5" event={"ID":"69d4933c-21c0-4d77-be02-395c15d250dc","Type":"ContainerDied","Data":"ecc66a6b1b5cf8d57dc65e9149d6316d4ac27d7f8ae31a1f70e8a7afe5e0d519"}
Oct 31 19:35:24.411577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.410874    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:35:24.416565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.412464    5958 scope.go:122] "RemoveContainer" containerID="519c95d263c6926409ed204550cd1ff3e40a823b80b682e31ff801a4766f8172"
Oct 31 19:35:24.549982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.549880    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.550479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.550448    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:35:24.568227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.568186    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.652623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.648004    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:35:24.806933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.776185    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded"
Oct 31 19:35:24.816564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.794586    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad60fd70-ea75-4b07-b330-b58ade511e54" path="/var/lib/kubelet/pods/ad60fd70-ea75-4b07-b330-b58ade511e54/volumes"
Oct 31 19:35:24.818020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:24.817989    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:35:25.432819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:25.432787    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:35:25.433387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:25.433367    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:35:25.509140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:25.509098    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:35:25.566578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:25.549290    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:25.610492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:25.610446    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 31 19:35:26.123621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.117097    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.133558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.133492    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.171871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.167286    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"]
Oct 31 19:35:26.343568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.335498    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ad60fd70-ea75-4b07-b330-b58ade511e54" containerName="payload-processing"
Oct 31 19:35:26.343568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.335578    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad60fd70-ea75-4b07-b330-b58ade511e54" containerName="payload-processing"
Oct 31 19:35:26.343568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.335661    5958 cpu_manager.go:414] "containerMap: removing container" podUID="69d4933c-21c0-4d77-be02-395c15d250dc" containerName="maas-api"
Oct 31 19:35:26.343568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.335672    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d4933c-21c0-4d77-be02-395c15d250dc" containerName="maas-api"
Oct 31 19:35:26.343568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.335876    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d4933c-21c0-4d77-be02-395c15d250dc" containerName="maas-api"
Oct 31 19:35:26.343568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.335904    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad60fd70-ea75-4b07-b330-b58ade511e54" containerName="payload-processing"
Oct 31 19:35:26.442577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.442461    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.468250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.468207    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.814369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.814250    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.899594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.838957    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.901181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.878832    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:35:26.920567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.917320    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.971632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:26.968950    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:27.013690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.013658    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.023322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.023292    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0b9ff685\""
Oct 31 19:35:27.073122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.073038    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0b9ff685-dockercfg-4lfnh\""
Oct 31 19:35:27.105569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.079978    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6vq\" (UniqueName: \"kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq\") pod \"payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") " pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.106361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.106315    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume\") pod \"payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") " pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.174449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.174403    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"]
Oct 31 19:35:27.174682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.174603    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"]
Oct 31 19:35:27.195753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.195697    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"]
Oct 31 19:35:27.196114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.196092    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.198715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.198672    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"]
Oct 31 19:35:27.241697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.241649    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume\") pod \"payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") " pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.242115 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.242090    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6vq\" (UniqueName: \"kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq\") pod \"payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") " pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.246663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.246628    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume\") pod \"payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") " pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.293302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.293270    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6vq\" (UniqueName: \"kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq\") pod \"payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") " pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.297404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.297374    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1cc6560c-67b565dbc8-hvwk5"]
Oct 31 19:35:27.346621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.345750    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.346621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.346013    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6k8\" (UniqueName: \"kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8\") pod \"payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.397369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.397334    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:27.463588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.454681    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.463588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.454899    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6k8\" (UniqueName: \"kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8\") pod \"payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.463588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.456397    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.494694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.483329    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6k8\" (UniqueName: \"kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8\") pod \"payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:27.648311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:27.648276    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:28.198585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:28.198229    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"]
Oct 31 19:35:28.201748 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:35:28.201717    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c93317_475a_4401_ae29_cc37561ce301.slice/crio-6612dea94bcb1608828303cb44bb0f30e6c94efad7d4b159f492e2ba8f997d62 WatchSource:0}: Error finding container 6612dea94bcb1608828303cb44bb0f30e6c94efad7d4b159f492e2ba8f997d62: Status 404 returned error can't find the container with id 6612dea94bcb1608828303cb44bb0f30e6c94efad7d4b159f492e2ba8f997d62
Oct 31 19:35:28.667111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:28.667082    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d4933c-21c0-4d77-be02-395c15d250dc" path="/var/lib/kubelet/pods/69d4933c-21c0-4d77-be02-395c15d250dc/volumes"
Oct 31 19:35:28.738634 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:35:28.738593    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c65cd4c_b0ed_4380_98d4_8119bbb4ac38.slice/crio-4f4aac3ef220d98cdf39e8d3824019f9c0b308ef3bcf8da6986e0c8c08e2d962 WatchSource:0}: Error finding container 4f4aac3ef220d98cdf39e8d3824019f9c0b308ef3bcf8da6986e0c8c08e2d962: Status 404 returned error can't find the container with id 4f4aac3ef220d98cdf39e8d3824019f9c0b308ef3bcf8da6986e0c8c08e2d962
Oct 31 19:35:28.739784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:28.739758    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"]
Oct 31 19:35:29.214485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.214445    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m" event={"ID":"47c93317-475a-4401-ae29-cc37561ce301","Type":"ContainerStarted","Data":"31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92"}
Oct 31 19:35:29.214698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.214582    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m" event={"ID":"47c93317-475a-4401-ae29-cc37561ce301","Type":"ContainerStarted","Data":"6612dea94bcb1608828303cb44bb0f30e6c94efad7d4b159f492e2ba8f997d62"}
Oct 31 19:35:29.214698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.214617    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:29.216803 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.216768    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k" event={"ID":"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38","Type":"ContainerStarted","Data":"e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1"}
Oct 31 19:35:29.217057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.216829    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k" event={"ID":"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38","Type":"ContainerStarted","Data":"4f4aac3ef220d98cdf39e8d3824019f9c0b308ef3bcf8da6986e0c8c08e2d962"}
Oct 31 19:35:29.217057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.216861    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:29.241792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.241735    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m" podStartSLOduration=5.241717276 podStartE2EDuration="5.241717276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:29.239085228 +0000 UTC m=+922.700459843" watchObservedRunningTime="2026-10-31 19:35:29.241717276 +0000 UTC m=+922.703091892"
Oct 31 19:35:29.271033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:29.270973    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k" podStartSLOduration=5.270954385 podStartE2EDuration="5.270954385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:29.263622279 +0000 UTC m=+922.724996903" watchObservedRunningTime="2026-10-31 19:35:29.270954385 +0000 UTC m=+922.732329001"
Oct 31 19:35:30.876936 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:30.876903    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"]
Oct 31 19:35:30.882577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:30.882537    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:30.894848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:30.894821    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-63eab93e-serving-cert\""
Oct 31 19:35:30.899480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:30.899439    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"]
Oct 31 19:35:31.001019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.000984    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.001194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.001039    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.001271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.001250    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x9s\" (UniqueName: \"kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.001305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.001288    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.124568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.122349    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.124568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.123160    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.124568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.123453    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x9s\" (UniqueName: \"kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.124568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.123525    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.125816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.125786    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.164614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.164406    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.166638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.166607    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x9s\" (UniqueName: \"kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.167534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.167375    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs\") pod \"maas-api-e2e-ait-63eab93e-65f7498b49-mr84w\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.216945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.216904    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:31.553451 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:35:31.553418    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f67895_9ff6_464d_af67_8383936ed9b5.slice/crio-e8a6dafa34dd44f3f2e184a3b4fb2c8a0755501338d83c430e008fa561b8cc0c WatchSource:0}: Error finding container e8a6dafa34dd44f3f2e184a3b4fb2c8a0755501338d83c430e008fa561b8cc0c: Status 404 returned error can't find the container with id e8a6dafa34dd44f3f2e184a3b4fb2c8a0755501338d83c430e008fa561b8cc0c
Oct 31 19:35:31.555229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:31.555204    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"]
Oct 31 19:35:32.289622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:32.289577    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w" event={"ID":"34f67895-9ff6-464d-af67-8383936ed9b5","Type":"ContainerStarted","Data":"f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a"}
Oct 31 19:35:32.290036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:32.289652    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:32.290036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:32.289673    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w" event={"ID":"34f67895-9ff6-464d-af67-8383936ed9b5","Type":"ContainerStarted","Data":"e8a6dafa34dd44f3f2e184a3b4fb2c8a0755501338d83c430e008fa561b8cc0c"}
Oct 31 19:35:32.316269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:32.316167    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w" podStartSLOduration=1.8225393890000001 podStartE2EDuration="2.316152505s" totalImagesPullingTime="493.613116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:32.311961482 +0000 UTC m=+925.773336100" watchObservedRunningTime="2026-10-31 19:35:32.316152505 +0000 UTC m=+925.777527119"
Oct 31 19:35:34.415098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.415039    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:35:34.788379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.782345    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"]
Oct 31 19:35:34.792238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.792205    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:34.794722 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.794681    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"]
Oct 31 19:35:34.799922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.799885    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0b9ff685-serving-cert\""
Oct 31 19:35:34.933573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.932715    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:34.933573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.932798    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:34.933573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.932928    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46hg\" (UniqueName: \"kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:34.933573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:34.933045    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.038662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.037628    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.038662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.037710    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.038662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.037829    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46hg\" (UniqueName: \"kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.038662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.037955    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.043843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.039980    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.050708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.047249    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.050708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.047784    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.057604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.055887    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46hg\" (UniqueName: \"kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg\") pod \"maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.112446 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.112411    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:35.560066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.547047    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"]
Oct 31 19:35:35.953916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:35.953889    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:35:36.430432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:36.430385    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz" event={"ID":"a54bc22d-041b-44c1-b12a-5a5fc3a09b33","Type":"ContainerStarted","Data":"722f94fba401a13a8d200032c5145e43d01b365c7a4d0c30c3ab82a0009e8adb"}
Oct 31 19:35:36.430649 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:36.430453    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz" event={"ID":"a54bc22d-041b-44c1-b12a-5a5fc3a09b33","Type":"ContainerStarted","Data":"62fe18ea545d2a6c0ea81ede24f7d2817abd7f81f23cd96a85b96e74244699e6"}
Oct 31 19:35:36.430649 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:36.430526    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:36.461870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:36.461797    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz" podStartSLOduration=1.946992169 podStartE2EDuration="2.461777215s" totalImagesPullingTime="514.785046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:36.459875832 +0000 UTC m=+929.921250450" watchObservedRunningTime="2026-10-31 19:35:36.461777215 +0000 UTC m=+929.923151832"
Oct 31 19:35:38.324439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:38.324407    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:35:40.257746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:40.257716    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:35:40.258233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:40.257841    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:35:42.480598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:42.480566    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:35:45.055727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:45.055653    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"]
Oct 31 19:35:45.081432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:45.081392    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820a1ecc-xvjkw"]
Oct 31 19:35:46.666195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:46.666150    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9ed52f-9c85-4647-a080-a8392a1c982b" path="/var/lib/kubelet/pods/0f9ed52f-9c85-4647-a080-a8392a1c982b/volumes"
Oct 31 19:35:47.150973 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:47.150899    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:35:54.493635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:54.493589    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"]
Oct 31 19:35:54.508523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:54.508486    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"]
Oct 31 19:35:57.384568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.383637    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:57.440569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.439633    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:35:57.471000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.470958    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"]
Oct 31 19:35:57.513117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.512618    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"]
Oct 31 19:35:57.680619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.651639    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:57.918578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.897644    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:57.967880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.967749    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:35:57.968217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:57.968174    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.085203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.084779    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.147568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.145622    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:35:58.371565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.368625    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:35:58.450741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.444740    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.480426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.451899    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:35:58.480840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.480779    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.543089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.543055    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:35:58.543713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:58.543675    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:35:59.157028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.156976    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.200772 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.200715    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:35:59.542977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.542868    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p" podUID="e0345914-4dac-4174-915d-d16fcd9decd0" containerName="payload-pre-processing" containerID="cri-o://392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc" gracePeriod=25
Oct 31 19:35:59.555986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.555930    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw" podUID="21d105c4-79ff-4690-b3d3-6fe001aa7ef2" containerName="payload-processing" containerID="cri-o://7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f" gracePeriod=25
Oct 31 19:35:59.633774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.633733    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.670951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.658367    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.679949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.678683    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:35:59.759021    5958 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.094s"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759081    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"]
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759219    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759261    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759294    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759328    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759363    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:35:59.763099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.759910    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w" podUID="34f67895-9ff6-464d-af67-8383936ed9b5" containerName="maas-api" containerID="cri-o://f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a" gracePeriod=30
Oct 31 19:35:59.947038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.946979    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.980773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:35:59.980714    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:00.161348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:00.161316    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:36:00.300896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:00.215881    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:36:00.300896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:00.277849    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:36:00.300896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:00.295838    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:00.807777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:00.711533    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.313013 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.299010    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.421014 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.420983    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:36:01.421283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.421265    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:36:01.424615 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:01.424159    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d105c4_79ff_4690_b3d3_6fe001aa7ef2.slice/crio-7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0345914_4dac_4174_915d_d16fcd9decd0.slice/crio-conmon-392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d105c4_79ff_4690_b3d3_6fe001aa7ef2.slice/crio-conmon-7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f67895_9ff6_464d_af67_8383936ed9b5.slice/crio-f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0345914_4dac_4174_915d_d16fcd9decd0.slice/crio-392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f67895_9ff6_464d_af67_8383936ed9b5.slice/crio-conmon-f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 19:36:01.524785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.524720    5958 generic.go:356] "Generic (PLEG): container finished" podID="21d105c4-79ff-4690-b3d3-6fe001aa7ef2" containerID="7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f" exitCode=0
Oct 31 19:36:01.525036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.524997    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw" event={"ID":"21d105c4-79ff-4690-b3d3-6fe001aa7ef2","Type":"ContainerDied","Data":"7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f"}
Oct 31 19:36:01.530740 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.530688    5958 generic.go:356] "Generic (PLEG): container finished" podID="e0345914-4dac-4174-915d-d16fcd9decd0" containerID="392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc" exitCode=0
Oct 31 19:36:01.531044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.531020    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p" event={"ID":"e0345914-4dac-4174-915d-d16fcd9decd0","Type":"ContainerDied","Data":"392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc"}
Oct 31 19:36:01.539231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.539183    5958 generic.go:356] "Generic (PLEG): container finished" podID="34f67895-9ff6-464d-af67-8383936ed9b5" containerID="f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a" exitCode=0
Oct 31 19:36:01.540494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.540463    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w" event={"ID":"34f67895-9ff6-464d-af67-8383936ed9b5","Type":"ContainerDied","Data":"f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a"}
Oct 31 19:36:01.607886 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.607762    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:36:01.719139 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.719092    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume\") pod \"e0345914-4dac-4174-915d-d16fcd9decd0\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") "
Oct 31 19:36:01.719419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.719235    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d\" (UniqueName: \"kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d\") pod \"e0345914-4dac-4174-915d-d16fcd9decd0\" (UID: \"e0345914-4dac-4174-915d-d16fcd9decd0\") "
Oct 31 19:36:01.736299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.735583    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume" pod "e0345914-4dac-4174-915d-d16fcd9decd0" (UID: "e0345914-4dac-4174-915d-d16fcd9decd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:01.763968 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.763903    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d" pod "e0345914-4dac-4174-915d-d16fcd9decd0" (UID: "e0345914-4dac-4174-915d-d16fcd9decd0"). InnerVolumeSpecName "kube-api-access-bwx9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:01.879754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.848405    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0345914-4dac-4174-915d-d16fcd9decd0-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:01.880326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.880295    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwx9d\" (UniqueName: \"kubernetes.io/projected/e0345914-4dac-4174-915d-d16fcd9decd0-kube-api-access-bwx9d\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:01.963566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.951726    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:36:01.982787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:01.982754    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:36:02.095608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.087664    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca\") pod \"34f67895-9ff6-464d-af67-8383936ed9b5\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") "
Oct 31 19:36:02.095608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.087986    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s\" (UniqueName: \"kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s\") pod \"34f67895-9ff6-464d-af67-8383936ed9b5\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") "
Oct 31 19:36:02.095608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.088046    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz\" (UniqueName: \"kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz\") pod \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") "
Oct 31 19:36:02.095608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.088083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume\") pod \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\" (UID: \"21d105c4-79ff-4690-b3d3-6fe001aa7ef2\") "
Oct 31 19:36:02.095608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.088154    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs\") pod \"34f67895-9ff6-464d-af67-8383936ed9b5\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") "
Oct 31 19:36:02.095608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.088232    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls\") pod \"34f67895-9ff6-464d-af67-8383936ed9b5\" (UID: \"34f67895-9ff6-464d-af67-8383936ed9b5\") "
Oct 31 19:36:02.109559 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.109507    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s" pod "34f67895-9ff6-464d-af67-8383936ed9b5" (UID: "34f67895-9ff6-464d-af67-8383936ed9b5"). InnerVolumeSpecName "kube-api-access-b5x9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:02.109907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.109885    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca" pod "34f67895-9ff6-464d-af67-8383936ed9b5" (UID: "34f67895-9ff6-464d-af67-8383936ed9b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:02.110397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.110368    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume" pod "21d105c4-79ff-4690-b3d3-6fe001aa7ef2" (UID: "21d105c4-79ff-4690-b3d3-6fe001aa7ef2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:02.110850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.110828    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5x9s\" (UniqueName: \"kubernetes.io/projected/34f67895-9ff6-464d-af67-8383936ed9b5-kube-api-access-b5x9s\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.110976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.110962    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.111061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.111050    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f67895-9ff6-464d-af67-8383936ed9b5-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.185828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.185282    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls" pod "34f67895-9ff6-464d-af67-8383936ed9b5" (UID: "34f67895-9ff6-464d-af67-8383936ed9b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:36:02.189825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.189794    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz" pod "21d105c4-79ff-4690-b3d3-6fe001aa7ef2" (UID: "21d105c4-79ff-4690-b3d3-6fe001aa7ef2"). InnerVolumeSpecName "kube-api-access-kmhxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:02.196225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.196152    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs" pod "34f67895-9ff6-464d-af67-8383936ed9b5" (UID: "34f67895-9ff6-464d-af67-8383936ed9b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:36:02.212999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.212960    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmhxz\" (UniqueName: \"kubernetes.io/projected/21d105c4-79ff-4690-b3d3-6fe001aa7ef2-kube-api-access-kmhxz\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.212999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.212996    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.212999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.213007    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34f67895-9ff6-464d-af67-8383936ed9b5-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.477654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.475198    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m" podUID="47c93317-475a-4401-ae29-cc37561ce301" containerName="payload-processing" containerID="cri-o://31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92" gracePeriod=25
Oct 31 19:36:02.516854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.515674    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k" podUID="5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" containerName="payload-pre-processing" containerID="cri-o://e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1" gracePeriod=25
Oct 31 19:36:02.560132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.560100    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"
Oct 31 19:36:02.560132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.560123    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w" event={"ID":"34f67895-9ff6-464d-af67-8383936ed9b5","Type":"ContainerDied","Data":"e8a6dafa34dd44f3f2e184a3b4fb2c8a0755501338d83c430e008fa561b8cc0c"}
Oct 31 19:36:02.560366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.560192    5958 scope.go:122] "RemoveContainer" containerID="f71266c5487e299aae1236945e84af81b0a8fd7ed798a0a6250f9228a5643f4a"
Oct 31 19:36:02.562242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.562221    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"
Oct 31 19:36:02.562364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.562253    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw" event={"ID":"21d105c4-79ff-4690-b3d3-6fe001aa7ef2","Type":"ContainerDied","Data":"e0bc1b66e6147856d3114ed4a08879a68eeba21d0fa691cc26f6d010e0c67fb6"}
Oct 31 19:36:02.563911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.563887    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p" event={"ID":"e0345914-4dac-4174-915d-d16fcd9decd0","Type":"ContainerDied","Data":"669006e05d5a95b04f74e93dd0874ab218dd079e7c6de313c298a1ef7ce314e8"}
Oct 31 19:36:02.563911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.563892    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"
Oct 31 19:36:02.641392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.641368    5958 scope.go:122] "RemoveContainer" containerID="7486269d4a096924592237aec56a5337a981faaabb1d12db315859eae38bf90f"
Oct 31 19:36:02.661869 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.661837    5958 scope.go:122] "RemoveContainer" containerID="392cd783d92139d699699f523c48cf9ec833099255dbc2bafea8918f8aa4a7bc"
Oct 31 19:36:02.664716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.663602    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"]
Oct 31 19:36:02.696010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.695978    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-63eab93e-8596d6f7ff-5pq8p"]
Oct 31 19:36:02.705597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.705469    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"]
Oct 31 19:36:02.712291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.712229    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-63eab93e-65f7498b49-mr84w"]
Oct 31 19:36:02.722145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.722104    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"]
Oct 31 19:36:02.741658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.741616    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-63eab93e-5bdf4b59f9-bsxqw"]
Oct 31 19:36:02.884835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.884800    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:36:02.898243 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.898214    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:36:02.957510 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.957480    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8\" (UniqueName: \"kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8\") pod \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") "
Oct 31 19:36:02.957702 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.957684    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume\") pod \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\" (UID: \"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38\") "
Oct 31 19:36:02.958054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.958030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume" pod "5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" (UID: "5c65cd4c-b0ed-4380-98d4-8119bbb4ac38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:02.958380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.958362    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:02.959839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:02.959810    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8" pod "5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" (UID: "5c65cd4c-b0ed-4380-98d4-8119bbb4ac38"). InnerVolumeSpecName "kube-api-access-lx6k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:03.059239 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.059157    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq\" (UniqueName: \"kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq\") pod \"47c93317-475a-4401-ae29-cc37561ce301\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") "
Oct 31 19:36:03.059471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.059449    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume\") pod \"47c93317-475a-4401-ae29-cc37561ce301\" (UID: \"47c93317-475a-4401-ae29-cc37561ce301\") "
Oct 31 19:36:03.059845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.059821    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume" pod "47c93317-475a-4401-ae29-cc37561ce301" (UID: "47c93317-475a-4401-ae29-cc37561ce301"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:03.060190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.060168    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c93317-475a-4401-ae29-cc37561ce301-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:03.060255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.060196    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6k8\" (UniqueName: \"kubernetes.io/projected/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38-kube-api-access-lx6k8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:03.061528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.061505    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq" pod "47c93317-475a-4401-ae29-cc37561ce301" (UID: "47c93317-475a-4401-ae29-cc37561ce301"). InnerVolumeSpecName "kube-api-access-6c6vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:03.162749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.162665    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6vq\" (UniqueName: \"kubernetes.io/projected/47c93317-475a-4401-ae29-cc37561ce301-kube-api-access-6c6vq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:03.576344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.576305    5958 generic.go:356] "Generic (PLEG): container finished" podID="5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" containerID="e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1" exitCode=0
Oct 31 19:36:03.576577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.576360    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"
Oct 31 19:36:03.576577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.576387    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k" event={"ID":"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38","Type":"ContainerDied","Data":"e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1"}
Oct 31 19:36:03.576577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.576456    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k" event={"ID":"5c65cd4c-b0ed-4380-98d4-8119bbb4ac38","Type":"ContainerDied","Data":"4f4aac3ef220d98cdf39e8d3824019f9c0b308ef3bcf8da6986e0c8c08e2d962"}
Oct 31 19:36:03.576577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.576486    5958 scope.go:122] "RemoveContainer" containerID="e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1"
Oct 31 19:36:03.580500 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.580459    5958 generic.go:356] "Generic (PLEG): container finished" podID="47c93317-475a-4401-ae29-cc37561ce301" containerID="31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92" exitCode=0
Oct 31 19:36:03.580670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.580513    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"
Oct 31 19:36:03.580670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.580588    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m" event={"ID":"47c93317-475a-4401-ae29-cc37561ce301","Type":"ContainerDied","Data":"31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92"}
Oct 31 19:36:03.580670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.580633    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m" event={"ID":"47c93317-475a-4401-ae29-cc37561ce301","Type":"ContainerDied","Data":"6612dea94bcb1608828303cb44bb0f30e6c94efad7d4b159f492e2ba8f997d62"}
Oct 31 19:36:03.591753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.591077    5958 scope.go:122] "RemoveContainer" containerID="e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1"
Oct 31 19:36:03.591753 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:03.591468    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1\": container with ID starting with e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1 not found: ID does not exist" containerID="e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1"
Oct 31 19:36:03.591753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.591504    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1"} err="failed to get container status \"e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1\": rpc error: code = NotFound desc = could not find container \"e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1\": container with ID starting with e68e992fa6cb01083dec6384e78171c5bf39cf31d9bf43b63850cc156c0a27e1 not found: ID does not exist"
Oct 31 19:36:03.591753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.591530    5958 scope.go:122] "RemoveContainer" containerID="31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92"
Oct 31 19:36:03.603020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.602989    5958 scope.go:122] "RemoveContainer" containerID="31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92"
Oct 31 19:36:03.603383 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:03.603361    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92\": container with ID starting with 31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92 not found: ID does not exist" containerID="31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92"
Oct 31 19:36:03.603466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.603395    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92"} err="failed to get container status \"31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92\": rpc error: code = NotFound desc = could not find container \"31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92\": container with ID starting with 31823ffe31f0c63e72834729082ce053a039eafceff4536eed433e712f567c92 not found: ID does not exist"
Oct 31 19:36:03.619925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.619886    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"]
Oct 31 19:36:03.630829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.630793    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0b9ff685-b95f54655-qk54m"]
Oct 31 19:36:03.735617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.735221    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"]
Oct 31 19:36:03.748732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:03.748575    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0b9ff685-85b47b7c79-wp72k"]
Oct 31 19:36:04.430840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.430803    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"]
Oct 31 19:36:04.431519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.431479    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz" podUID="a54bc22d-041b-44c1-b12a-5a5fc3a09b33" containerName="maas-api" containerID="cri-o://722f94fba401a13a8d200032c5145e43d01b365c7a4d0c30c3ab82a0009e8adb" gracePeriod=30
Oct 31 19:36:04.596969 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.596936    5958 generic.go:356] "Generic (PLEG): container finished" podID="a54bc22d-041b-44c1-b12a-5a5fc3a09b33" containerID="722f94fba401a13a8d200032c5145e43d01b365c7a4d0c30c3ab82a0009e8adb" exitCode=0
Oct 31 19:36:04.597980 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.597073    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz" event={"ID":"a54bc22d-041b-44c1-b12a-5a5fc3a09b33","Type":"ContainerDied","Data":"722f94fba401a13a8d200032c5145e43d01b365c7a4d0c30c3ab82a0009e8adb"}
Oct 31 19:36:04.670235 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.670197    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d105c4-79ff-4690-b3d3-6fe001aa7ef2" path="/var/lib/kubelet/pods/21d105c4-79ff-4690-b3d3-6fe001aa7ef2/volumes"
Oct 31 19:36:04.670998 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.670970    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f67895-9ff6-464d-af67-8383936ed9b5" path="/var/lib/kubelet/pods/34f67895-9ff6-464d-af67-8383936ed9b5/volumes"
Oct 31 19:36:04.671622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.671575    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c93317-475a-4401-ae29-cc37561ce301" path="/var/lib/kubelet/pods/47c93317-475a-4401-ae29-cc37561ce301/volumes"
Oct 31 19:36:04.672773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.672747    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" path="/var/lib/kubelet/pods/5c65cd4c-b0ed-4380-98d4-8119bbb4ac38/volumes"
Oct 31 19:36:04.673369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.673347    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0345914-4dac-4174-915d-d16fcd9decd0" path="/var/lib/kubelet/pods/e0345914-4dac-4174-915d-d16fcd9decd0/volumes"
Oct 31 19:36:04.735531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.735502    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:36:04.939569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.933579    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls\") pod \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") "
Oct 31 19:36:04.939569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.933851    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca\") pod \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") "
Oct 31 19:36:04.939569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.933968    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs\") pod \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") "
Oct 31 19:36:04.939569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.934054    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg\" (UniqueName: \"kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg\") pod \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\" (UID: \"a54bc22d-041b-44c1-b12a-5a5fc3a09b33\") "
Oct 31 19:36:04.946802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.946681    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca" pod "a54bc22d-041b-44c1-b12a-5a5fc3a09b33" (UID: "a54bc22d-041b-44c1-b12a-5a5fc3a09b33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:04.973926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:04.973882    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs" pod "a54bc22d-041b-44c1-b12a-5a5fc3a09b33" (UID: "a54bc22d-041b-44c1-b12a-5a5fc3a09b33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:36:05.050567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.044139    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:05.050567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.044186    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:05.106564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.100935    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg" pod "a54bc22d-041b-44c1-b12a-5a5fc3a09b33" (UID: "a54bc22d-041b-44c1-b12a-5a5fc3a09b33"). InnerVolumeSpecName "kube-api-access-q46hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:05.126779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.111938    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls" pod "a54bc22d-041b-44c1-b12a-5a5fc3a09b33" (UID: "a54bc22d-041b-44c1-b12a-5a5fc3a09b33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:36:05.185073 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.159016    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46hg\" (UniqueName: \"kubernetes.io/projected/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-kube-api-access-q46hg\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:05.185073 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.159058    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a54bc22d-041b-44c1-b12a-5a5fc3a09b33-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:05.638242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.638206    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz" event={"ID":"a54bc22d-041b-44c1-b12a-5a5fc3a09b33","Type":"ContainerDied","Data":"62fe18ea545d2a6c0ea81ede24f7d2817abd7f81f23cd96a85b96e74244699e6"}
Oct 31 19:36:05.638728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.638302    5958 scope.go:122] "RemoveContainer" containerID="722f94fba401a13a8d200032c5145e43d01b365c7a4d0c30c3ab82a0009e8adb"
Oct 31 19:36:05.638728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.638439    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"
Oct 31 19:36:05.688146 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.688100    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"]
Oct 31 19:36:05.744572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:05.744397    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0b9ff685-549b5f794b-sz6hz"]
Oct 31 19:36:06.278382 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.278344    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"]
Oct 31 19:36:06.279203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279182    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" containerName="payload-pre-processing"
Oct 31 19:36:06.279262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279206    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" containerName="payload-pre-processing"
Oct 31 19:36:06.279262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279228    5958 cpu_manager.go:414] "containerMap: removing container" podUID="21d105c4-79ff-4690-b3d3-6fe001aa7ef2" containerName="payload-processing"
Oct 31 19:36:06.279262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279237    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d105c4-79ff-4690-b3d3-6fe001aa7ef2" containerName="payload-processing"
Oct 31 19:36:06.279262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279256    5958 cpu_manager.go:414] "containerMap: removing container" podUID="34f67895-9ff6-464d-af67-8383936ed9b5" containerName="maas-api"
Oct 31 19:36:06.279262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279263    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f67895-9ff6-464d-af67-8383936ed9b5" containerName="maas-api"
Oct 31 19:36:06.279431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279295    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e0345914-4dac-4174-915d-d16fcd9decd0" containerName="payload-pre-processing"
Oct 31 19:36:06.279431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279302    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0345914-4dac-4174-915d-d16fcd9decd0" containerName="payload-pre-processing"
Oct 31 19:36:06.279431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279311    5958 cpu_manager.go:414] "containerMap: removing container" podUID="47c93317-475a-4401-ae29-cc37561ce301" containerName="payload-processing"
Oct 31 19:36:06.279431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279318    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c93317-475a-4401-ae29-cc37561ce301" containerName="payload-processing"
Oct 31 19:36:06.279431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279339    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a54bc22d-041b-44c1-b12a-5a5fc3a09b33" containerName="maas-api"
Oct 31 19:36:06.279431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279346    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54bc22d-041b-44c1-b12a-5a5fc3a09b33" containerName="maas-api"
Oct 31 19:36:06.279640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279524    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c65cd4c-b0ed-4380-98d4-8119bbb4ac38" containerName="payload-pre-processing"
Oct 31 19:36:06.279640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279538    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d105c4-79ff-4690-b3d3-6fe001aa7ef2" containerName="payload-processing"
Oct 31 19:36:06.279640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279562    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f67895-9ff6-464d-af67-8383936ed9b5" containerName="maas-api"
Oct 31 19:36:06.279640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279575    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0345914-4dac-4174-915d-d16fcd9decd0" containerName="payload-pre-processing"
Oct 31 19:36:06.279640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279584    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c93317-475a-4401-ae29-cc37561ce301" containerName="payload-processing"
Oct 31 19:36:06.279640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.279594    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54bc22d-041b-44c1-b12a-5a5fc3a09b33" containerName="maas-api"
Oct 31 19:36:06.290724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.290685    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"]
Oct 31 19:36:06.290866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.290838    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.295984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.295951    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-a891ce-openshift-default-dockercfg-t8vjd\""
Oct 31 19:36:06.409171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409120    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409186    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409281    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409373    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409438    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq7p\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409602 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409532    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409604    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.409710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409687    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.417574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.409780    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.532810    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.532883    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.532970    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.533039    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.533128    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.533164    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.533268    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.533413    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.536611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.533483    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq7p\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.540568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.538911    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.559565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.552144    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.559565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.552392    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.559565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.552628    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.559565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.552829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.571566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.567842    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.595124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.595050    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq7p\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.596048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.595989    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.609467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.608140    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.627808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.611085    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:06.764569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:06.759511    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54bc22d-041b-44c1-b12a-5a5fc3a09b33" path="/var/lib/kubelet/pods/a54bc22d-041b-44c1-b12a-5a5fc3a09b33/volumes"
Oct 31 19:36:07.165696 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:07.165663    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fa028a_9dc6_4935_9456_1dc22bfb2578.slice/crio-a1b7aaa06ecaed13b8c8398cdb5de5132ccf472fb6e0a10a530bca81bcfb79e4 WatchSource:0}: Error finding container a1b7aaa06ecaed13b8c8398cdb5de5132ccf472fb6e0a10a530bca81bcfb79e4: Status 404 returned error can't find the container with id a1b7aaa06ecaed13b8c8398cdb5de5132ccf472fb6e0a10a530bca81bcfb79e4
Oct 31 19:36:07.166225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.166145    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"]
Oct 31 19:36:07.168269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.168231    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:36:07.168370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.168310    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:36:07.168370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.168353    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:36:07.745845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.745797    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" event={"ID":"e1fa028a-9dc6-4935-9456-1dc22bfb2578","Type":"ContainerStarted","Data":"e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"}
Oct 31 19:36:07.746047 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.745874    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" event={"ID":"e1fa028a-9dc6-4935-9456-1dc22bfb2578","Type":"ContainerStarted","Data":"a1b7aaa06ecaed13b8c8398cdb5de5132ccf472fb6e0a10a530bca81bcfb79e4"}
Oct 31 19:36:07.774414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:07.774352    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podStartSLOduration=1.774338339 podStartE2EDuration="1.774338339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:07.772236539 +0000 UTC m=+961.233611154" watchObservedRunningTime="2026-10-31 19:36:07.774338339 +0000 UTC m=+961.235712954"
Oct 31 19:36:08.627182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:08.627144    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:08.633216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:08.633187    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:08.832630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:08.830846    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:08.832630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:08.832306    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:36:09.305157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.305127    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"]
Oct 31 19:36:09.310486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.310461    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.323751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.323726    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-cd9729-dockercfg-2nfqx\""
Oct 31 19:36:09.326506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.326479    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-cd9729\""
Oct 31 19:36:09.328413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.328389    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"]
Oct 31 19:36:09.356302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.356264    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"]
Oct 31 19:36:09.360986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.360961    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.384486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.384453    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"]
Oct 31 19:36:09.432567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.426131    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfddw\" (UniqueName: \"kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw\") pod \"payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.432567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.426288    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.432567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.426335    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.432567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.426389    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nm84\" (UniqueName: \"kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84\") pod \"payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.533568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.533177    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfddw\" (UniqueName: \"kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw\") pod \"payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.533568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.533306    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.533568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.533349    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.533568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.533399    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nm84\" (UniqueName: \"kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84\") pod \"payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.540570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.535019    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.577220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.546015    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.580567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.578311    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nm84\" (UniqueName: \"kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84\") pod \"payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.584903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.584538    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfddw\" (UniqueName: \"kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw\") pod \"payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:09.641713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.641677    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:09.698566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:09.697287    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:10.101748 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:10.101705    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072664a6_4f1d_4318_b9e4_b31190d80f64.slice/crio-28193b7917516a854e186ab1d69394ed55038ed73466bb2ed6cb9a5889afc069 WatchSource:0}: Error finding container 28193b7917516a854e186ab1d69394ed55038ed73466bb2ed6cb9a5889afc069: Status 404 returned error can't find the container with id 28193b7917516a854e186ab1d69394ed55038ed73466bb2ed6cb9a5889afc069
Oct 31 19:36:10.102293 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.102270    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"]
Oct 31 19:36:10.129747 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:10.129712    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae17531_6e12_4803_9977_cb2e9a349df5.slice/crio-55402883db2f733fb536ee3f43fde10269404cf629375f1565421baa61e3b412 WatchSource:0}: Error finding container 55402883db2f733fb536ee3f43fde10269404cf629375f1565421baa61e3b412: Status 404 returned error can't find the container with id 55402883db2f733fb536ee3f43fde10269404cf629375f1565421baa61e3b412
Oct 31 19:36:10.130825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.130797    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"]
Oct 31 19:36:10.856912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.856864    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" event={"ID":"4ae17531-6e12-4803-9977-cb2e9a349df5","Type":"ContainerStarted","Data":"eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f"}
Oct 31 19:36:10.857126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.856928    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" event={"ID":"4ae17531-6e12-4803-9977-cb2e9a349df5","Type":"ContainerStarted","Data":"55402883db2f733fb536ee3f43fde10269404cf629375f1565421baa61e3b412"}
Oct 31 19:36:10.857126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.857063    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:10.859588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.859510    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" event={"ID":"072664a6-4f1d-4318-b9e4-b31190d80f64","Type":"ContainerStarted","Data":"7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590"}
Oct 31 19:36:10.859588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.859583    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" event={"ID":"072664a6-4f1d-4318-b9e4-b31190d80f64","Type":"ContainerStarted","Data":"28193b7917516a854e186ab1d69394ed55038ed73466bb2ed6cb9a5889afc069"}
Oct 31 19:36:10.859750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.859619    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:10.881977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.881918    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" podStartSLOduration=1.881903047 podStartE2EDuration="1.881903047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:10.878521104 +0000 UTC m=+964.339895731" watchObservedRunningTime="2026-10-31 19:36:10.881903047 +0000 UTC m=+964.343277662"
Oct 31 19:36:10.907182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:10.907130    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" podStartSLOduration=1.907116814 podStartE2EDuration="1.907116814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:10.906812957 +0000 UTC m=+964.368187571" watchObservedRunningTime="2026-10-31 19:36:10.907116814 +0000 UTC m=+964.368491428"
Oct 31 19:36:11.935373 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.935337    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"]
Oct 31 19:36:11.940321 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.940299    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:11.944711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.944686    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"]
Oct 31 19:36:11.947354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.947338    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-a891ce\""
Oct 31 19:36:11.949506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.949229    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-a891ce-dockercfg-djmgw\""
Oct 31 19:36:11.949702 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.949685    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:11.959385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.959359    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"]
Oct 31 19:36:11.966201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:11.966171    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"]
Oct 31 19:36:12.029497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.029424    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.029695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.029521    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmfg\" (UniqueName: \"kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg\") pod \"payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.029776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.029701    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.029776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.029767    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9c2n\" (UniqueName: \"kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n\") pod \"payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.182569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.171882    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.182569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.171976    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9c2n\" (UniqueName: \"kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n\") pod \"payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.182569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.172208    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.182569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.172276    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmfg\" (UniqueName: \"kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg\") pod \"payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.182569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.177790    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.182569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.177927    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.253619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.253525    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9c2n\" (UniqueName: \"kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n\") pod \"payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.274603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.263780    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmfg\" (UniqueName: \"kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg\") pod \"payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.274603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.270239    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:12.560108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.560017    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:12.753325 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:12.753289    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5602cf8f_d9ef_412b_8901_91285525c030.slice/crio-4c9e7b2f4ceab7740f1acf25ac19dbf3ff5a9ea66380a62212b269605373e5c9 WatchSource:0}: Error finding container 4c9e7b2f4ceab7740f1acf25ac19dbf3ff5a9ea66380a62212b269605373e5c9: Status 404 returned error can't find the container with id 4c9e7b2f4ceab7740f1acf25ac19dbf3ff5a9ea66380a62212b269605373e5c9
Oct 31 19:36:12.753862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.753817    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"]
Oct 31 19:36:12.930302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.927657    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"]
Oct 31 19:36:12.951626 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:12.947354    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4347d253_53c6_4a1b_80a1_83a92f0202d0.slice/crio-a656abcc4f9a1496aed7f4b441276d62df81ec69c22f1552acf7bd412f4e88a9 WatchSource:0}: Error finding container a656abcc4f9a1496aed7f4b441276d62df81ec69c22f1552acf7bd412f4e88a9: Status 404 returned error can't find the container with id a656abcc4f9a1496aed7f4b441276d62df81ec69c22f1552acf7bd412f4e88a9
Oct 31 19:36:12.998818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.995242    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" event={"ID":"5602cf8f-d9ef-412b-8901-91285525c030","Type":"ContainerStarted","Data":"e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3"}
Oct 31 19:36:12.998818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.995330    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" event={"ID":"5602cf8f-d9ef-412b-8901-91285525c030","Type":"ContainerStarted","Data":"4c9e7b2f4ceab7740f1acf25ac19dbf3ff5a9ea66380a62212b269605373e5c9"}
Oct 31 19:36:12.998818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:12.996703    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:13.046584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:13.046507    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" podStartSLOduration=2.046492032 podStartE2EDuration="2.046492032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:13.041399046 +0000 UTC m=+966.502773673" watchObservedRunningTime="2026-10-31 19:36:13.046492032 +0000 UTC m=+966.507866647"
Oct 31 19:36:14.005771 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:14.005707    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" event={"ID":"4347d253-53c6-4a1b-80a1-83a92f0202d0","Type":"ContainerStarted","Data":"cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e"}
Oct 31 19:36:14.006227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:14.005822    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" event={"ID":"4347d253-53c6-4a1b-80a1-83a92f0202d0","Type":"ContainerStarted","Data":"a656abcc4f9a1496aed7f4b441276d62df81ec69c22f1552acf7bd412f4e88a9"}
Oct 31 19:36:14.006227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:14.006079    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:14.030743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:14.030687    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" podStartSLOduration=3.030671884 podStartE2EDuration="3.030671884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:14.025714293 +0000 UTC m=+967.487088941" watchObservedRunningTime="2026-10-31 19:36:14.030671884 +0000 UTC m=+967.492046499"
Oct 31 19:36:15.911587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:15.911532    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"]
Oct 31 19:36:15.921412 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:15.917642    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:15.925118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:15.924250    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-cd9729-serving-cert\""
Oct 31 19:36:15.940161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:15.940123    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"]
Oct 31 19:36:16.078295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.078198    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslf7\" (UniqueName: \"kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.078295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.078266    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.078295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.078298    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.078836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.078386    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.086518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.086482    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"]
Oct 31 19:36:16.099949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.098702    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41294a12-zclff"]
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.180953    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslf7\" (UniqueName: \"kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.186906    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.186951    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.187043    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:16.187420    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-serving-cert: secret "maas-api-e2e-shared-a-w1-cd9729-serving-cert" not found
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:16.187496    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls podName:128d8a3d-d166-482f-9e12-00ebf955e68b nodeName:}" failed. No retries permitted until 2026-10-31 19:36:16.687471842 +0000 UTC m=+970.148846461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls") pod "maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" (UID: "128d8a3d-d166-482f-9e12-00ebf955e68b") : secret "maas-api-e2e-shared-a-w1-cd9729-serving-cert" not found
Oct 31 19:36:16.188656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.188586    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.192857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.192824    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.199202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.198972    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslf7\" (UniqueName: \"kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.712361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.711731    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.801929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.800373    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0822c1-32d7-4098-8248-191261d34c1e" path="/var/lib/kubelet/pods/ad0822c1-32d7-4098-8248-191261d34c1e/volumes"
Oct 31 19:36:16.830256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.814079    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:16.853407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:16.853372    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:17.048180 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:17.048067    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:36:17.167942 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:17.167907    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128d8a3d_d166_482f_9e12_00ebf955e68b.slice/crio-68d0c4945404770711ff3745e038c33bf9d9959b974c909555a9d12ea405f603 WatchSource:0}: Error finding container 68d0c4945404770711ff3745e038c33bf9d9959b974c909555a9d12ea405f603: Status 404 returned error can't find the container with id 68d0c4945404770711ff3745e038c33bf9d9959b974c909555a9d12ea405f603
Oct 31 19:36:17.169324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:17.169301    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"]
Oct 31 19:36:18.035729 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.035660    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"]
Oct 31 19:36:18.040690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.040654    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c82f2c83-zhhgj"]
Oct 31 19:36:18.045408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.045361    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" event={"ID":"128d8a3d-d166-482f-9e12-00ebf955e68b","Type":"ContainerStarted","Data":"3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030"}
Oct 31 19:36:18.045600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.045441    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" event={"ID":"128d8a3d-d166-482f-9e12-00ebf955e68b","Type":"ContainerStarted","Data":"68d0c4945404770711ff3745e038c33bf9d9959b974c909555a9d12ea405f603"}
Oct 31 19:36:18.045600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.045472    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:18.071177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.071128    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" podStartSLOduration=2.590044991 podStartE2EDuration="3.071113589s" totalImagesPullingTime="481.068598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:18.067463364 +0000 UTC m=+971.528837993" watchObservedRunningTime="2026-10-31 19:36:18.071113589 +0000 UTC m=+971.532488218"
Oct 31 19:36:18.666440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.666407    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42555d54-d741-4ca3-8b29-97f9ffc4368c" path="/var/lib/kubelet/pods/42555d54-d741-4ca3-8b29-97f9ffc4368c/volumes"
Oct 31 19:36:18.793250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.793218    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"]
Oct 31 19:36:18.798982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.798950    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.813969 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.813832    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-a891ce-serving-cert\""
Oct 31 19:36:18.815685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.814217    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"]
Oct 31 19:36:18.862191 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.862146    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.862360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.862259    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.862360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.862329    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxcwv\" (UniqueName: \"kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.862492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.862474    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.963610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.963465    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.963868 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.963846    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.963928 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.963911    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.964024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.964008    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxcwv\" (UniqueName: \"kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.964697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.964671    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.966412 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.966386    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:18.966674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:18.966649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:19.038572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:19.035335    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxcwv\" (UniqueName: \"kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv\") pod \"maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:19.135937 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:19.135898    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:19.539557 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:19.539246    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"]
Oct 31 19:36:20.073199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:20.073163    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x" event={"ID":"c04b6288-9859-4958-9bc1-4104c779a292","Type":"ContainerStarted","Data":"fe868af85208e9edc824c05e7d3d44f76c572e72581944248aae97cea5fbb995"}
Oct 31 19:36:21.036414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.036361    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"]
Oct 31 19:36:21.041169 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.041138    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a642b2ad-x85ck"]
Oct 31 19:36:21.083448 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.083408    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x" event={"ID":"c04b6288-9859-4958-9bc1-4104c779a292","Type":"ContainerStarted","Data":"f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64"}
Oct 31 19:36:21.083635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.083502    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:21.110697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.110646    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x" podStartSLOduration=2.587546139 podStartE2EDuration="3.110630634s" totalImagesPullingTime="523.084495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:21.106652642 +0000 UTC m=+974.568027257" watchObservedRunningTime="2026-10-31 19:36:21.110630634 +0000 UTC m=+974.572005274"
Oct 31 19:36:21.872307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.872280    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 19:36:21.872500 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:21.872356    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 19:36:22.666179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:22.666141    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f656db-5838-4262-9a2b-6c5519738fcf" path="/var/lib/kubelet/pods/c7f656db-5838-4262-9a2b-6c5519738fcf/volumes"
Oct 31 19:36:24.009623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:24.009592    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 19:36:24.067419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:24.067385    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 19:36:25.030008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:25.029982    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 19:36:27.098864 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:27.098836    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 19:36:28.360566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:28.354672    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:36:30.033696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:30.033655    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"]
Oct 31 19:36:30.038473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:30.038437    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc48236-sgvph"]
Oct 31 19:36:30.665845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:30.665810    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f044f0-71d8-406f-9ac0-4f9fa6ccb32c" path="/var/lib/kubelet/pods/68f044f0-71d8-406f-9ac0-4f9fa6ccb32c/volumes"
Oct 31 19:36:34.694776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.694746    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"]
Oct 31 19:36:34.701450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.701426    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"]
Oct 31 19:36:34.701613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.701517    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:34.707351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.707325    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-435728\""
Oct 31 19:36:34.715820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.715795    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-435728-dockercfg-zfsp9\""
Oct 31 19:36:34.724001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.723971    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"]
Oct 31 19:36:34.729260 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.729233    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:34.768064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.767930    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:34.779959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.779896    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:34.783307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.783268    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmcq\" (UniqueName: \"kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq\") pod \"payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:34.809904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.809861    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzb89\" (UniqueName: \"kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89\") pod \"payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:34.844992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.844955    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"]
Oct 31 19:36:34.956156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:34.956067    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:35.162466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.157818    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmcq\" (UniqueName: \"kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq\") pod \"payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:35.162466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.158037    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzb89\" (UniqueName: \"kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89\") pod \"payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:35.162466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.158135    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:35.162466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.159460    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:35.162466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.078478    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:35.176654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.176149    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmcq\" (UniqueName: \"kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq\") pod \"payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:35.177423 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.177391    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzb89\" (UniqueName: \"kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89\") pod \"payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:35.336115 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.336028    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:35.356179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.356146    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:35.755998 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:35.755964    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbe721a_665b_4c89_b288_0d295946bba3.slice/crio-dc7aefabd163db05090cf5bf36905b13c0f0bcb6b0eafeb286265fee94fdf5cd WatchSource:0}: Error finding container dc7aefabd163db05090cf5bf36905b13c0f0bcb6b0eafeb286265fee94fdf5cd: Status 404 returned error can't find the container with id dc7aefabd163db05090cf5bf36905b13c0f0bcb6b0eafeb286265fee94fdf5cd
Oct 31 19:36:35.756406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.756010    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"]
Oct 31 19:36:35.761350 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:35.761321    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b0ef0f_bc50_4197_a382_0c4df79be7ee.slice/crio-4e9725542a167934f472f9c32fb6d99469a720a71411c4f13a5d3bf02548aaa3 WatchSource:0}: Error finding container 4e9725542a167934f472f9c32fb6d99469a720a71411c4f13a5d3bf02548aaa3: Status 404 returned error can't find the container with id 4e9725542a167934f472f9c32fb6d99469a720a71411c4f13a5d3bf02548aaa3
Oct 31 19:36:35.761467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:35.761360    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"]
Oct 31 19:36:36.032168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.032063    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"]
Oct 31 19:36:36.036878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.036848    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9441f3-lsxqc"]
Oct 31 19:36:36.414318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.414284    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" event={"ID":"f4b0ef0f-bc50-4197-a382-0c4df79be7ee","Type":"ContainerStarted","Data":"af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c"}
Oct 31 19:36:36.414509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.414366    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" event={"ID":"f4b0ef0f-bc50-4197-a382-0c4df79be7ee","Type":"ContainerStarted","Data":"4e9725542a167934f472f9c32fb6d99469a720a71411c4f13a5d3bf02548aaa3"}
Oct 31 19:36:36.414509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.414416    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:36.416262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.416237    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" event={"ID":"efbe721a-665b-4c89-b288-0d295946bba3","Type":"ContainerStarted","Data":"d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f"}
Oct 31 19:36:36.416378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.416277    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" event={"ID":"efbe721a-665b-4c89-b288-0d295946bba3","Type":"ContainerStarted","Data":"dc7aefabd163db05090cf5bf36905b13c0f0bcb6b0eafeb286265fee94fdf5cd"}
Oct 31 19:36:36.416378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.416325    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:36.435121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.435066    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" podStartSLOduration=2.435053881 podStartE2EDuration="2.435053881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:36.434602166 +0000 UTC m=+989.895976781" watchObservedRunningTime="2026-10-31 19:36:36.435053881 +0000 UTC m=+989.896428495"
Oct 31 19:36:36.459632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.459576    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" podStartSLOduration=2.459560701 podStartE2EDuration="2.459560701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:36.457407536 +0000 UTC m=+989.918782154" watchObservedRunningTime="2026-10-31 19:36:36.459560701 +0000 UTC m=+989.920935313"
Oct 31 19:36:36.666966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.666893    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44bae8b8-1627-49a5-962e-83a0d12dbfe2" path="/var/lib/kubelet/pods/44bae8b8-1627-49a5-962e-83a0d12dbfe2/volumes"
Oct 31 19:36:36.693637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.693603    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"]
Oct 31 19:36:36.701345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.701321    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:36.707921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.707894    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-e1d060\""
Oct 31 19:36:36.708128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.708102    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"]
Oct 31 19:36:36.708279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.708188    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-e1d060-dockercfg-5z7m6\""
Oct 31 19:36:36.750380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.750345    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"]
Oct 31 19:36:36.755784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.755756    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:36.778062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.778028    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"]
Oct 31 19:36:36.812842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.812801    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:36.813611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.813574    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw96d\" (UniqueName: \"kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d\") pod \"payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:36.951621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.951279    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw96d\" (UniqueName: \"kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d\") pod \"payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:36.951621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.951343    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjts8\" (UniqueName: \"kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8\") pod \"payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:36.951621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.951516    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:36.952173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.951792    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:37.015802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:36.961166    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:37.026639 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.024136    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw96d\" (UniqueName: \"kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d\") pod \"payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:37.058626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.054043    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjts8\" (UniqueName: \"kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8\") pod \"payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:37.058626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.054373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:37.058626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.055715    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:37.151134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.133081    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjts8\" (UniqueName: \"kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8\") pod \"payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:37.322951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.322867    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:37.378591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.377980    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:37.676696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.675011    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"]
Oct 31 19:36:37.739892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:37.739864    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"]
Oct 31 19:36:38.485074 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.485033    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" event={"ID":"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b","Type":"ContainerStarted","Data":"f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2"}
Oct 31 19:36:38.485621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.485099    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" event={"ID":"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b","Type":"ContainerStarted","Data":"141d4101fead9a3495d85428ea44e3abc30793702e0de54300240f92a3bb57e0"}
Oct 31 19:36:38.485621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.485142    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:38.487253 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.487226    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" event={"ID":"b326360d-426f-4051-8d04-7856458d4950","Type":"ContainerStarted","Data":"b1426ee79149ff92c8e03e69e5daf7470727ebe3eb1e0301de0f0daaac8cda43"}
Oct 31 19:36:38.487364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.487288    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" event={"ID":"b326360d-426f-4051-8d04-7856458d4950","Type":"ContainerStarted","Data":"5cc4f0bb7a2190693fea817c961a4d7b6862e2759d45a911468c4ef134ff9e16"}
Oct 31 19:36:38.487697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.487677    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:38.507912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.507860    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" podStartSLOduration=2.507845071 podStartE2EDuration="2.507845071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:38.50633795 +0000 UTC m=+991.967712564" watchObservedRunningTime="2026-10-31 19:36:38.507845071 +0000 UTC m=+991.969219685"
Oct 31 19:36:38.543810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:38.543745    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" podStartSLOduration=2.5437287189999997 podStartE2EDuration="2.543728719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:38.52709144 +0000 UTC m=+991.988466056" watchObservedRunningTime="2026-10-31 19:36:38.543728719 +0000 UTC m=+992.005103355"
Oct 31 19:36:42.173299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.173264    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"]
Oct 31 19:36:42.179073 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.179041    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.194388 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.194352    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-435728-serving-cert\""
Oct 31 19:36:42.198499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.198473    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"]
Oct 31 19:36:42.245204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.245167    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.245386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.245340    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.245386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.245371    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjwf\" (UniqueName: \"kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.245525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.245502    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.357565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.356965    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.357565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.357039    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjwf\" (UniqueName: \"kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.357565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.357128    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.357565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.357375    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.364627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.358345    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.378564 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:42.376138    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-serving-cert: secret "maas-api-e2e-shared-b-w1-435728-serving-cert" not found
Oct 31 19:36:42.378564 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:42.376218    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls podName:3d88c070-0722-481d-9056-1f2aa89a02a3 nodeName:}" failed. No retries permitted until 2026-10-31 19:36:42.876196212 +0000 UTC m=+996.337570806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls") pod "maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" (UID: "3d88c070-0722-481d-9056-1f2aa89a02a3") : secret "maas-api-e2e-shared-b-w1-435728-serving-cert" not found
Oct 31 19:36:42.381683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.381647    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.397583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.397534    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjwf\" (UniqueName: \"kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.889572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.886309    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:42.930568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:42.926677    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:43.044847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:43.044801    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"]
Oct 31 19:36:43.050720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:43.050693    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e5464bb-sdp2f"]
Oct 31 19:36:43.104691 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:43.104656    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:43.407578 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:43.407490    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d88c070_0722_481d_9056_1f2aa89a02a3.slice/crio-4cf34c6f3868f0693b09b8e15a58cc9129a84f27d9d3ebff58ab54670e6157d5 WatchSource:0}: Error finding container 4cf34c6f3868f0693b09b8e15a58cc9129a84f27d9d3ebff58ab54670e6157d5: Status 404 returned error can't find the container with id 4cf34c6f3868f0693b09b8e15a58cc9129a84f27d9d3ebff58ab54670e6157d5
Oct 31 19:36:43.408658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:43.408533    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"]
Oct 31 19:36:43.595379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:43.595289    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" event={"ID":"3d88c070-0722-481d-9056-1f2aa89a02a3","Type":"ContainerStarted","Data":"4cf34c6f3868f0693b09b8e15a58cc9129a84f27d9d3ebff58ab54670e6157d5"}
Oct 31 19:36:44.605394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:44.605350    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" event={"ID":"3d88c070-0722-481d-9056-1f2aa89a02a3","Type":"ContainerStarted","Data":"e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085"}
Oct 31 19:36:44.605884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:44.605423    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:44.634084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:44.634028    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" podStartSLOduration=2.198349083 podStartE2EDuration="2.634012404s" totalImagesPullingTime="435.663321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:44.627678013 +0000 UTC m=+998.089052629" watchObservedRunningTime="2026-10-31 19:36:44.634012404 +0000 UTC m=+998.095387018"
Oct 31 19:36:44.667213 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:44.667164    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ba162b-c822-47c4-9923-9bc5430de637" path="/var/lib/kubelet/pods/c7ba162b-c822-47c4-9923-9bc5430de637/volumes"
Oct 31 19:36:45.185701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.185663    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"]
Oct 31 19:36:45.191195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.191170    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.220490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.220460    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-e1d060-serving-cert\""
Oct 31 19:36:45.225583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.225537    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"]
Oct 31 19:36:45.272035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.271997    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.272238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.272125    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.272299 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.272252    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.272335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.272297    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgmz\" (UniqueName: \"kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.383357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.383313    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.383596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.383564    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.383738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.383720    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.383805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.383786    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgmz\" (UniqueName: \"kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.385176 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.385141    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.388960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.387933    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.388960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.388450    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.434565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.428326    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgmz\" (UniqueName: \"kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz\") pod \"maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.530265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.530175    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:45.818910 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:36:45.818869    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d3b347_992f_4d11_8358_1af3997f684e.slice/crio-3d2ffe6d1fa8331ba14805f7e648453f9fce5eef6dc65c42d4ae8773e4f0d13b WatchSource:0}: Error finding container 3d2ffe6d1fa8331ba14805f7e648453f9fce5eef6dc65c42d4ae8773e4f0d13b: Status 404 returned error can't find the container with id 3d2ffe6d1fa8331ba14805f7e648453f9fce5eef6dc65c42d4ae8773e4f0d13b
Oct 31 19:36:45.821334 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:45.821306    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"]
Oct 31 19:36:46.626794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:46.626756    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx" event={"ID":"61d3b347-992f-4d11-8358-1af3997f684e","Type":"ContainerStarted","Data":"89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205"}
Oct 31 19:36:46.626983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:46.626814    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx" event={"ID":"61d3b347-992f-4d11-8358-1af3997f684e","Type":"ContainerStarted","Data":"3d2ffe6d1fa8331ba14805f7e648453f9fce5eef6dc65c42d4ae8773e4f0d13b"}
Oct 31 19:36:46.626983 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:46.626902    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:36:46.651307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:46.651247    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx" podStartSLOduration=1.192243652 podStartE2EDuration="1.651232531s" totalImagesPullingTime="458.988879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:46.649390298 +0000 UTC m=+1000.110764936" watchObservedRunningTime="2026-10-31 19:36:46.651232531 +0000 UTC m=+1000.112607145"
Oct 31 19:36:47.048712 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:36:47.048616    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:36:47.458597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:47.451683    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 19:36:47.458597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:47.456267    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 19:36:49.507220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:49.507190    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 19:36:49.507798 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:49.507258    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 19:36:50.707233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:50.707207    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 19:36:52.647090 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:36:52.647063    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 19:37:01.423569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:01.418889    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:37:01.423569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:01.418970    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:07.044861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.044825    5958 scope.go:122] "RemoveContainer" containerID="8872822e1faaed006f6e385f03ec60e7635cebce8dc28028cfc94682085324f2"
Oct 31 19:37:07.540842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.540808    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"]
Oct 31 19:37:07.550592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.548048    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.553678 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.553648    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"]
Oct 31 19:37:07.564565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.562815    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-67adab47\""
Oct 31 19:37:07.705566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.704763    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.705566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.705112    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtwz\" (UniqueName: \"kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.817621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.807731    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtwz\" (UniqueName: \"kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.817621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.807925    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.817621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.809076    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.862760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.859276    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtwz\" (UniqueName: \"kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:07.892572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:07.887894    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:08.376788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:08.376756    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"]
Oct 31 19:37:08.377492 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:37:08.377457    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39eb1bf_7616_4ff7_a27a_9314360604dd.slice/crio-ade50530e779e0328a42b8f77a29f29bf6153bcea05ba4eb1013d75c371a57c5 WatchSource:0}: Error finding container ade50530e779e0328a42b8f77a29f29bf6153bcea05ba4eb1013d75c371a57c5: Status 404 returned error can't find the container with id ade50530e779e0328a42b8f77a29f29bf6153bcea05ba4eb1013d75c371a57c5
Oct 31 19:37:09.292232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:09.291707    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" event={"ID":"e39eb1bf-7616-4ff7-a27a-9314360604dd","Type":"ContainerStarted","Data":"3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480"}
Oct 31 19:37:09.292232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:09.291784    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" event={"ID":"e39eb1bf-7616-4ff7-a27a-9314360604dd","Type":"ContainerStarted","Data":"ade50530e779e0328a42b8f77a29f29bf6153bcea05ba4eb1013d75c371a57c5"}
Oct 31 19:37:09.294633 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:09.293141    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:09.348973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:09.348904    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" podStartSLOduration=2.348884846 podStartE2EDuration="2.348884846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:09.345223539 +0000 UTC m=+1022.806598154" watchObservedRunningTime="2026-10-31 19:37:09.348884846 +0000 UTC m=+1022.810259468"
Oct 31 19:37:17.203929 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:37:17.203855    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:37:21.385720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:21.380356    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:37:31.740000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:37:31.739966    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:37:47.047560 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:37:47.047512    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:38:03.039645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:03.039612    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"]
Oct 31 19:38:03.044140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:03.044113    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b246870-7vlr8"]
Oct 31 19:38:04.228622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:04.228589    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:38:04.667203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:04.667170    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a81b0d-b809-4fa1-8721-b6b24ceedda5" path="/var/lib/kubelet/pods/a0a81b0d-b809-4fa1-8721-b6b24ceedda5/volumes"
Oct 31 19:38:07.103475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:07.103431    5958 scope.go:122] "RemoveContainer" containerID="58828fb24eb5117526a7300f5b654b7a61ed352984c4be10b58cc62c993aba0c"
Oct 31 19:38:07.195135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:07.195104    5958 scope.go:122] "RemoveContainer" containerID="720216432c80e4c06c406dc7bff50e9d664423d3cebea34d6f5a050b4a6a880e"
Oct 31 19:38:07.284826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:07.279836    5958 scope.go:122] "RemoveContainer" containerID="fcae6c84635f7926d9e5cf320382e052cceb42ebfeecd862069cb32aca3c63cc"
Oct 31 19:38:07.295340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:07.295318    5958 scope.go:122] "RemoveContainer" containerID="57f947eac4e203acee9ec1e92f8b4b0ac02374753225755118378c9cfa650c30"
Oct 31 19:38:07.354874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:07.354788    5958 scope.go:122] "RemoveContainer" containerID="b09779af4f1fe7cba2c5482f4a0f1626f5b86bc5ae6bbad63b6103f52b048e81"
Oct 31 19:38:07.397401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:07.397371    5958 scope.go:122] "RemoveContainer" containerID="ca543181671a8ba8c49849a1e5186e840cdbbfcc32e321ef5bb593d4ab87f9d1"
Oct 31 19:38:09.081853 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:09.081813    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"]
Oct 31 19:38:09.094964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:09.094925    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ea630f0-zqkfj"]
Oct 31 19:38:10.667643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:10.667606    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260ea7a5-8029-4c62-9ed5-801c264f7819" path="/var/lib/kubelet/pods/260ea7a5-8029-4c62-9ed5-801c264f7819/volumes"
Oct 31 19:38:17.051677 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:38:17.051594    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:38:19.380867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.380833    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"]
Oct 31 19:38:19.388777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.388751    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.397076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.397040    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-db61066b\""
Oct 31 19:38:19.397231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.397169    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"]
Oct 31 19:38:19.428990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.428948    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"]
Oct 31 19:38:19.434265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.434239    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:19.444298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.444268    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-db61066b-dockercfg-5n48c\""
Oct 31 19:38:19.455924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.455888    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"]
Oct 31 19:38:19.473217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.473177    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.473460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.473412    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsjpx\" (UniqueName: \"kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx\") pod \"payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.652637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.635203    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.652637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.635413    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume\") pod \"payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") " pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:19.652637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.635469    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsjpx\" (UniqueName: \"kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx\") pod \"payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.652637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.638815    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.652996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.652683    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrzr\" (UniqueName: \"kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr\") pod \"payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") " pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:19.762996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.759870    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume\") pod \"payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") " pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:19.768123 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.763435    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrzr\" (UniqueName: \"kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr\") pod \"payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") " pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:19.770957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.766228    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsjpx\" (UniqueName: \"kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx\") pod \"payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:19.772058 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.769009    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume\") pod \"payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") " pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:19.874115 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:19.871736    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrzr\" (UniqueName: \"kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr\") pod \"payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") " pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:20.009458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:20.009374    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:20.061887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:20.061839    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:20.362890 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:38:20.362858    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4f944d_327a_4e39_82c3_79a90a0d7077.slice/crio-3489aa1412e2e7c7de9df2e9704efa6688c2e2407f10af8ba98be0358466f5ca WatchSource:0}: Error finding container 3489aa1412e2e7c7de9df2e9704efa6688c2e2407f10af8ba98be0358466f5ca: Status 404 returned error can't find the container with id 3489aa1412e2e7c7de9df2e9704efa6688c2e2407f10af8ba98be0358466f5ca
Oct 31 19:38:20.364090 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:20.364064    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"]
Oct 31 19:38:20.405222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:20.405196    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"]
Oct 31 19:38:20.408700 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:38:20.408665    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1497f0_b3ab_4453_a5e8_7d83b3e75f6c.slice/crio-96121a1c59785a1937f414a3f23eb15a29cfcb085c123c2aa3078d567df0fe89 WatchSource:0}: Error finding container 96121a1c59785a1937f414a3f23eb15a29cfcb085c123c2aa3078d567df0fe89: Status 404 returned error can't find the container with id 96121a1c59785a1937f414a3f23eb15a29cfcb085c123c2aa3078d567df0fe89
Oct 31 19:38:20.408836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:20.408701    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr" event={"ID":"ab4f944d-327a-4e39-82c3-79a90a0d7077","Type":"ContainerStarted","Data":"3489aa1412e2e7c7de9df2e9704efa6688c2e2407f10af8ba98be0358466f5ca"}
Oct 31 19:38:21.496638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.493267    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr" event={"ID":"ab4f944d-327a-4e39-82c3-79a90a0d7077","Type":"ContainerStarted","Data":"cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d"}
Oct 31 19:38:21.496638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.494655    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:21.519569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.512500    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7" event={"ID":"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c","Type":"ContainerStarted","Data":"fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4"}
Oct 31 19:38:21.519569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.512679    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7" event={"ID":"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c","Type":"ContainerStarted","Data":"96121a1c59785a1937f414a3f23eb15a29cfcb085c123c2aa3078d567df0fe89"}
Oct 31 19:38:21.519569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.513649    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:21.590840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.586410    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7" podStartSLOduration=2.586254173 podStartE2EDuration="2.586254173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:21.550571924 +0000 UTC m=+1095.011946540" watchObservedRunningTime="2026-10-31 19:38:21.586254173 +0000 UTC m=+1095.047628788"
Oct 31 19:38:21.590840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:21.586737    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr" podStartSLOduration=2.586724309 podStartE2EDuration="2.586724309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:21.525991221 +0000 UTC m=+1094.987365874" watchObservedRunningTime="2026-10-31 19:38:21.586724309 +0000 UTC m=+1095.048098924"
Oct 31 19:38:24.491207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.491168    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"]
Oct 31 19:38:24.496430 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.496400    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:24.500677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.500648    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"]
Oct 31 19:38:24.505676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.505651    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.508700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.508674    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9e323f\""
Oct 31 19:38:24.514350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.514317    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"]
Oct 31 19:38:24.523962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.523741    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9e323f-dockercfg-5bzcj\""
Oct 31 19:38:24.536500 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.533677    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"]
Oct 31 19:38:24.639716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.638860    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.639716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.638996    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fstr\" (UniqueName: \"kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr\") pod \"payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:24.639716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.639054    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:24.639716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.639218    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k54\" (UniqueName: \"kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54\") pod \"payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.766679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.746730    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:24.766679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.746926    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k54\" (UniqueName: \"kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54\") pod \"payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.766679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.747209    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.766679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.747316    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fstr\" (UniqueName: \"kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr\") pod \"payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:24.766679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.748405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.766679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.748405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:24.833329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.833299    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k54\" (UniqueName: \"kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54\") pod \"payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") " pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.842883 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.842843    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:24.853250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:24.853209    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fstr\" (UniqueName: \"kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr\") pod \"payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:25.118762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.117869    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:25.306300 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.305923    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"]
Oct 31 19:38:25.490158 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:38:25.490122    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba3e96_a367_438c_816d_8e119418a22a.slice/crio-963fc78e5b06b0be282102c8b1535b15dcbe9849631390b961141d6762dad4d3 WatchSource:0}: Error finding container 963fc78e5b06b0be282102c8b1535b15dcbe9849631390b961141d6762dad4d3: Status 404 returned error can't find the container with id 963fc78e5b06b0be282102c8b1535b15dcbe9849631390b961141d6762dad4d3
Oct 31 19:38:25.490363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.490345    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"]
Oct 31 19:38:25.714211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.714162    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62" event={"ID":"0410f536-f2eb-42bc-801f-c8a744546ebf","Type":"ContainerStarted","Data":"f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710"}
Oct 31 19:38:25.714924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.714230    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62" event={"ID":"0410f536-f2eb-42bc-801f-c8a744546ebf","Type":"ContainerStarted","Data":"d954daa15927f0979cbb0aae0613e863bd288b3784d1d877fff657092a98a50a"}
Oct 31 19:38:25.714924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.714253    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:25.716012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.715979    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx" event={"ID":"ceba3e96-a367-438c-816d-8e119418a22a","Type":"ContainerStarted","Data":"963fc78e5b06b0be282102c8b1535b15dcbe9849631390b961141d6762dad4d3"}
Oct 31 19:38:25.746136 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:25.746067    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62" podStartSLOduration=1.7460518600000001 podStartE2EDuration="1.74605186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:25.743827351 +0000 UTC m=+1099.205201966" watchObservedRunningTime="2026-10-31 19:38:25.74605186 +0000 UTC m=+1099.207426475"
Oct 31 19:38:26.729000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:26.728960    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx" event={"ID":"ceba3e96-a367-438c-816d-8e119418a22a","Type":"ContainerStarted","Data":"dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184"}
Oct 31 19:38:26.729404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:26.729053    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:26.760732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:26.760682    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx" podStartSLOduration=2.760667583 podStartE2EDuration="2.760667583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:26.7583306 +0000 UTC m=+1100.219705216" watchObservedRunningTime="2026-10-31 19:38:26.760667583 +0000 UTC m=+1100.222042197"
Oct 31 19:38:27.389197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.389152    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"]
Oct 31 19:38:27.394863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.394837    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.401754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.401729    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4bac5a-dockercfg-zlbv2\""
Oct 31 19:38:27.402949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.402926    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4bac5a\""
Oct 31 19:38:27.411102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.411073    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"]
Oct 31 19:38:27.495664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.495624    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zpk\" (UniqueName: \"kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk\") pod \"payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.495860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.495837    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.598307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.598267    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.616849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.616814    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zpk\" (UniqueName: \"kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk\") pod \"payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.617618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.599533    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.646627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.646381    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zpk\" (UniqueName: \"kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk\") pod \"payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:27.734846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:27.734809    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:28.209824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:28.209794    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"]
Oct 31 19:38:28.211750 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:38:28.211711    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fa35d0_5438_4dca_9ebe_ee270792971b.slice/crio-e41f4b6444b00c63d582772de5b7406dc2a0b3049519a0da5eec3ab80df58373 WatchSource:0}: Error finding container e41f4b6444b00c63d582772de5b7406dc2a0b3049519a0da5eec3ab80df58373: Status 404 returned error can't find the container with id e41f4b6444b00c63d582772de5b7406dc2a0b3049519a0da5eec3ab80df58373
Oct 31 19:38:28.888473 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:28.888411    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" event={"ID":"d6fa35d0-5438-4dca-9ebe-ee270792971b","Type":"ContainerStarted","Data":"b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561"}
Oct 31 19:38:28.889188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:28.889161    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" event={"ID":"d6fa35d0-5438-4dca-9ebe-ee270792971b","Type":"ContainerStarted","Data":"e41f4b6444b00c63d582772de5b7406dc2a0b3049519a0da5eec3ab80df58373"}
Oct 31 19:38:28.937238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:28.937181    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" podStartSLOduration=1.937167305 podStartE2EDuration="1.937167305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:28.930539569 +0000 UTC m=+1102.391914183" watchObservedRunningTime="2026-10-31 19:38:28.937167305 +0000 UTC m=+1102.398541920"
Oct 31 19:38:29.970624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:29.970589    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:32.034992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:32.034952    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"]
Oct 31 19:38:32.039712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:32.039684    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74e78f23-4hxsd"]
Oct 31 19:38:32.667689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:32.667654    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc82231-ff48-493e-bae1-b7ece14675fd" path="/var/lib/kubelet/pods/3bc82231-ff48-493e-bae1-b7ece14675fd/volumes"
Oct 31 19:38:33.579624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:33.566956    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:38:33.582686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:33.582659    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:38:36.756568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:36.753807    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:38:37.478660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.478625    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"]
Oct 31 19:38:37.485305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.485274    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.493184 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.493157    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9e323f-serving-cert\""
Oct 31 19:38:37.501034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.501008    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"]
Oct 31 19:38:37.644479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.644444    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.644695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.644521    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjttk\" (UniqueName: \"kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.644695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.644669    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.644917 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.644901    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.752616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.747472    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.752616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.747611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjttk\" (UniqueName: \"kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.752616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.747666    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.752616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.747981    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.780608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.778557    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.782295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.781456    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.790802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.788256    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.806393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.806362    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjttk\" (UniqueName: \"kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk\") pod \"maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.833933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.833891    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:37.874891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:37.874846    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:38:38.198875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:38.198046    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"]
Oct 31 19:38:39.110004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:39.109903    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" event={"ID":"0c477b14-5390-42fd-a54b-5a5ba5e6fa27","Type":"ContainerStarted","Data":"a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74"}
Oct 31 19:38:39.110004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:39.109966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" event={"ID":"0c477b14-5390-42fd-a54b-5a5ba5e6fa27","Type":"ContainerStarted","Data":"a9688cb87b7fa764037fbe3ff1f5fbb3551339c519dd9b58b1000d546d2e8566"}
Oct 31 19:38:39.110004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:39.109997    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:39.146124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:39.146071    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" podStartSLOduration=1.520474407 podStartE2EDuration="2.146054989s" totalImagesPullingTime="625.580582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:39.144803611 +0000 UTC m=+1112.606178228" watchObservedRunningTime="2026-10-31 19:38:39.146054989 +0000 UTC m=+1112.607429604"
Oct 31 19:38:40.303957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.303918    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"]
Oct 31 19:38:40.309676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.309654    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.321188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.321155    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4bac5a-serving-cert\""
Oct 31 19:38:40.329789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.329762    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"]
Oct 31 19:38:40.426041 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.426007    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.426237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.426156    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.426310 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.426293    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.426365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.426351    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtpr\" (UniqueName: \"kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.531584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.529944    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtpr\" (UniqueName: \"kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.531584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.530268    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.531584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.530412    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.531584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.530515    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.554628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.548809    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.554628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.549214    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.557966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.557896    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.564596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.564567    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtpr\" (UniqueName: \"kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr\") pod \"maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.639436 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.639392    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:40.976014 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.975986    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 19:38:40.996362 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:38:40.996327    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40045daf_6b06_40e0_90ee_aac67220d7b7.slice/crio-5425d6b18981505554acc8dc74063fe555a0c66f7c58584a422f84ab3e020b34 WatchSource:0}: Error finding container 5425d6b18981505554acc8dc74063fe555a0c66f7c58584a422f84ab3e020b34: Status 404 returned error can't find the container with id 5425d6b18981505554acc8dc74063fe555a0c66f7c58584a422f84ab3e020b34
Oct 31 19:38:40.997282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:40.997257    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"]
Oct 31 19:38:41.210573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:41.192300    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk" event={"ID":"40045daf-6b06-40e0-90ee-aac67220d7b7","Type":"ContainerStarted","Data":"5425d6b18981505554acc8dc74063fe555a0c66f7c58584a422f84ab3e020b34"}
Oct 31 19:38:42.203241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:42.203200    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk" event={"ID":"40045daf-6b06-40e0-90ee-aac67220d7b7","Type":"ContainerStarted","Data":"709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af"}
Oct 31 19:38:42.203750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:42.203304    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:42.229157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:42.229102    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk" podStartSLOduration=1.635488738 podStartE2EDuration="2.229088956s" totalImagesPullingTime="593.600218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:42.2286929 +0000 UTC m=+1115.690067515" watchObservedRunningTime="2026-10-31 19:38:42.229088956 +0000 UTC m=+1115.690463602"
Oct 31 19:38:45.127623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:45.127593    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:38:47.048760 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:38:47.048722    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:38:48.161309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:48.161273    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:38:48.309568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:48.306726    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 19:38:50.073323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:50.073286    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"]
Oct 31 19:38:50.077642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:50.077513    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53342951-hz6rh"]
Oct 31 19:38:50.711952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:50.711911    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0212e2e-6893-4d65-bfb7-0b410d4e4607" path="/var/lib/kubelet/pods/d0212e2e-6893-4d65-bfb7-0b410d4e4607/volumes"
Oct 31 19:38:54.029688 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:54.029652    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"]
Oct 31 19:38:54.034180 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:54.034151    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c59481-8pp6q"]
Oct 31 19:38:54.678232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:54.677411    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e99a89-3c6b-423e-8b69-604048be0f79" path="/var/lib/kubelet/pods/f9e99a89-3c6b-423e-8b69-604048be0f79/volumes"
Oct 31 19:38:56.045982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:56.045892    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"]
Oct 31 19:38:56.059518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:56.059189    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1fc859-zmvmj"]
Oct 31 19:38:56.503195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:56.503164    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"]
Oct 31 19:38:56.669675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:38:56.669632    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e42f63-a3ae-4d6b-b868-86dfba23ffe3" path="/var/lib/kubelet/pods/f4e42f63-a3ae-4d6b-b868-86dfba23ffe3/volumes"
Oct 31 19:39:01.513566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:01.508629    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerName="payload-processing" containerID="cri-o://3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480" gracePeriod=25
Oct 31 19:39:03.031433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:03.031400    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"]
Oct 31 19:39:03.035756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:03.035728    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-698f142d-wgdc9"]
Oct 31 19:39:04.669085 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:04.669044    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9890e04-0403-4496-abf7-4d19d34acb42" path="/var/lib/kubelet/pods/f9890e04-0403-4496-abf7-4d19d34acb42/volumes"
Oct 31 19:39:07.854845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.854810    5958 scope.go:122] "RemoveContainer" containerID="69b7f2c3687c88e68674e6104873e8c7c4386d970fd32b0e6e8be8c0858a53da"
Oct 31 19:39:07.871578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.870476    5958 scope.go:122] "RemoveContainer" containerID="174192f790fb4156510f5d9e18c0b1d26c10d632618e6f2a46d0a8ea537617ff"
Oct 31 19:39:07.910165 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.909804    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"]
Oct 31 19:39:07.918106 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.918076    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"]
Oct 31 19:39:07.918266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.918185    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:07.927834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.927797    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:07.932421 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.932397    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ae2e60-dockercfg-sj8hw\""
Oct 31 19:39:07.933379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.933359    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ae2e60\""
Oct 31 19:39:07.937471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.937448    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"]
Oct 31 19:39:07.959652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:07.959623    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"]
Oct 31 19:39:08.054571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.049273    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.054571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.049394    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhwb\" (UniqueName: \"kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb\") pod \"payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.054571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.049509    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snf5r\" (UniqueName: \"kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r\") pod \"payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.054571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.049692    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.304564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.302558    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhwb\" (UniqueName: \"kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb\") pod \"payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.304564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.302710    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snf5r\" (UniqueName: \"kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r\") pod \"payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.304564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.302868    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.304564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.303106    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.304564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.304050    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.334683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.305132    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.385436 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.385400    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snf5r\" (UniqueName: \"kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r\") pod \"payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.385699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.385502    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhwb\" (UniqueName: \"kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb\") pod \"payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.549658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.549625    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:08.571222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.571186    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:08.853496 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:39:08.853468    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd6fe3c_12bb_44c2_bd63_5a35ccd15162.slice/crio-7d67eb87243bad24f5ed4c2a57acb9f60389f907e033bd23c26168aed71a95be WatchSource:0}: Error finding container 7d67eb87243bad24f5ed4c2a57acb9f60389f907e033bd23c26168aed71a95be: Status 404 returned error can't find the container with id 7d67eb87243bad24f5ed4c2a57acb9f60389f907e033bd23c26168aed71a95be
Oct 31 19:39:08.854458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.854432    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"]
Oct 31 19:39:08.871860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.871831    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" event={"ID":"afd6fe3c-12bb-44c2-bd63-5a35ccd15162","Type":"ContainerStarted","Data":"7d67eb87243bad24f5ed4c2a57acb9f60389f907e033bd23c26168aed71a95be"}
Oct 31 19:39:08.873482 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:39:08.873457    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4aba490_0fe0_4097_9c2e_f74c4527706a.slice/crio-0256f021eb63cdc024fdf82c8af01bbbc4bce15424ce7e21a7f40eca7f599cce WatchSource:0}: Error finding container 0256f021eb63cdc024fdf82c8af01bbbc4bce15424ce7e21a7f40eca7f599cce: Status 404 returned error can't find the container with id 0256f021eb63cdc024fdf82c8af01bbbc4bce15424ce7e21a7f40eca7f599cce
Oct 31 19:39:08.873858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:08.873836    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"]
Oct 31 19:39:09.908938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.908903    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" event={"ID":"f4aba490-0fe0-4097-9c2e-f74c4527706a","Type":"ContainerStarted","Data":"002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325"}
Oct 31 19:39:09.909341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.908960    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" event={"ID":"f4aba490-0fe0-4097-9c2e-f74c4527706a","Type":"ContainerStarted","Data":"0256f021eb63cdc024fdf82c8af01bbbc4bce15424ce7e21a7f40eca7f599cce"}
Oct 31 19:39:09.909341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.908993    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:09.911157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.911128    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" event={"ID":"afd6fe3c-12bb-44c2-bd63-5a35ccd15162","Type":"ContainerStarted","Data":"9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a"}
Oct 31 19:39:09.911315 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.911187    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:09.935325 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.935270    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" podStartSLOduration=2.935251068 podStartE2EDuration="2.935251068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:09.929099404 +0000 UTC m=+1143.390474032" watchObservedRunningTime="2026-10-31 19:39:09.935251068 +0000 UTC m=+1143.396625688"
Oct 31 19:39:09.952296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:09.952243    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" podStartSLOduration=2.952230529 podStartE2EDuration="2.952230529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:09.948771245 +0000 UTC m=+1143.410145861" watchObservedRunningTime="2026-10-31 19:39:09.952230529 +0000 UTC m=+1143.413605144"
Oct 31 19:39:11.328369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:11.328328    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:17.077571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:17.074208    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"]
Oct 31 19:39:17.082714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:17.082502    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67129660-b9tmd"]
Oct 31 19:39:17.146458 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:39:17.146416    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:39:18.668082 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.668043    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c928f55f-fdf6-4616-8559-964634333fca" path="/var/lib/kubelet/pods/c928f55f-fdf6-4616-8559-964634333fca/volumes"
Oct 31 19:39:18.890791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.890741    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"]
Oct 31 19:39:18.901486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.901447    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:18.910510 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.910479    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d4b21c-serving-cert\""
Oct 31 19:39:18.913755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.913670    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"]
Oct 31 19:39:18.964927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.964895    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:18.965126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.965059    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:18.965232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.965207    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:18.965322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:18.965302    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6b7\" (UniqueName: \"kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.067503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.067467    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.067720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.067633    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.067794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.067724    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.067794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.067764    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6b7\" (UniqueName: \"kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.068398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.068371    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.070040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.070016    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.070163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.070139    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.083374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.083342    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6b7\" (UniqueName: \"kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7\") pod \"maas-api-e2e-api-b-d4b21c-6999489686-5pd7l\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.250800 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.242446    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:19.769573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:19.768187    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"]
Oct 31 19:39:20.033553 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:20.033449    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"]
Oct 31 19:39:20.039465 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:20.039434    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f32bd03c-tk5lf"]
Oct 31 19:39:20.177887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:20.177567    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" event={"ID":"f077dc9b-353f-4590-9cf5-4d406aaca326","Type":"ContainerStarted","Data":"576339f2ea9892ad3580f85199f61afc9d2ae692301c5e01b3139b0a9c2ee990"}
Oct 31 19:39:20.667513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:20.667475    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66c20c4-6181-4c46-861a-490262869859" path="/var/lib/kubelet/pods/f66c20c4-6181-4c46-861a-490262869859/volumes"
Oct 31 19:39:20.923374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:20.923294    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 19:39:20.923374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:20.923367    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 19:39:21.240629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.238668    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" event={"ID":"f077dc9b-353f-4590-9cf5-4d406aaca326","Type":"ContainerStarted","Data":"5dc3235c2227045ad6d5ef200d20847f6d2fd0f3f73a633deb0d2ef5d9c55e3e"}
Oct 31 19:39:21.240629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.239124    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:21.265014 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.264965    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" podStartSLOduration=2.724884123 podStartE2EDuration="3.26495149s" totalImagesPullingTime="540.067367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:21.263744865 +0000 UTC m=+1154.725119494" watchObservedRunningTime="2026-10-31 19:39:21.26495149 +0000 UTC m=+1154.726326104"
Oct 31 19:39:21.327517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.327472    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:21.842652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.842626    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:39:21.946295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.946251    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz\" (UniqueName: \"kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz\") pod \"e39eb1bf-7616-4ff7-a27a-9314360604dd\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") "
Oct 31 19:39:21.946841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.946434    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume\") pod \"e39eb1bf-7616-4ff7-a27a-9314360604dd\" (UID: \"e39eb1bf-7616-4ff7-a27a-9314360604dd\") "
Oct 31 19:39:21.947219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.947190    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume" pod "e39eb1bf-7616-4ff7-a27a-9314360604dd" (UID: "e39eb1bf-7616-4ff7-a27a-9314360604dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:21.947626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.947601    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39eb1bf-7616-4ff7-a27a-9314360604dd-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:21.948918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:21.948894    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz" pod "e39eb1bf-7616-4ff7-a27a-9314360604dd" (UID: "e39eb1bf-7616-4ff7-a27a-9314360604dd"). InnerVolumeSpecName "kube-api-access-xrtwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:22.049818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.049731    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtwz\" (UniqueName: \"kubernetes.io/projected/e39eb1bf-7616-4ff7-a27a-9314360604dd-kube-api-access-xrtwz\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.158503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.158463    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"]
Oct 31 19:39:22.159577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.159519    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerName="payload-processing"
Oct 31 19:39:22.159577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.159557    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerName="payload-processing"
Oct 31 19:39:22.159760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.159749    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerName="payload-processing"
Oct 31 19:39:22.177573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.176928    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.188173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.188139    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"]
Oct 31 19:39:22.195951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.195468    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ae2e60-serving-cert\""
Oct 31 19:39:22.251158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.251125    5958 generic.go:356] "Generic (PLEG): container finished" podID="e39eb1bf-7616-4ff7-a27a-9314360604dd" containerID="3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480" exitCode=0
Oct 31 19:39:22.251381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.251183    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"
Oct 31 19:39:22.251381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.251249    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" event={"ID":"e39eb1bf-7616-4ff7-a27a-9314360604dd","Type":"ContainerDied","Data":"3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480"}
Oct 31 19:39:22.251381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.251295    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7" event={"ID":"e39eb1bf-7616-4ff7-a27a-9314360604dd","Type":"ContainerDied","Data":"ade50530e779e0328a42b8f77a29f29bf6153bcea05ba4eb1013d75c371a57c5"}
Oct 31 19:39:22.251381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.251322    5958 scope.go:122] "RemoveContainer" containerID="3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480"
Oct 31 19:39:22.253072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.253047    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.253396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.253372    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.253658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.253515    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.253768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.253650    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7v7h\" (UniqueName: \"kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.265148 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.265124    5958 scope.go:122] "RemoveContainer" containerID="3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480"
Oct 31 19:39:22.265486 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:39:22.265466    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480\": container with ID starting with 3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480 not found: ID does not exist" containerID="3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480"
Oct 31 19:39:22.265628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.265494    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480"} err="failed to get container status \"3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480\": rpc error: code = NotFound desc = could not find container \"3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480\": container with ID starting with 3aade06dbd7e5c9dc84bba6228431f9648205686a0ebefd149e039f6fb386480 not found: ID does not exist"
Oct 31 19:39:22.343775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.339346    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"]
Oct 31 19:39:22.356640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.356450    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.359221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.356898    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.359221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.356942    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7v7h\" (UniqueName: \"kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.359221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.357211    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.359221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.358633    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.361033 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:39:22.360612    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-serving-cert: secret "maas-api-e2e-shared-b-w2-ae2e60-serving-cert" not found
Oct 31 19:39:22.361033 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:39:22.360691    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls podName:0d76eb2a-ecc5-4b45-a134-754216e355c0 nodeName:}" failed. No retries permitted until 2026-10-31 19:39:22.860666361 +0000 UTC m=+1156.322040954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" (UID: "0d76eb2a-ecc5-4b45-a134-754216e355c0") : secret "maas-api-e2e-shared-b-w2-ae2e60-serving-cert" not found
Oct 31 19:39:22.362167 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.362141    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-nnms7"]
Oct 31 19:39:22.374458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.374425    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.394470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.394383    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7v7h\" (UniqueName: \"kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.666773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.666739    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39eb1bf-7616-4ff7-a27a-9314360604dd" path="/var/lib/kubelet/pods/e39eb1bf-7616-4ff7-a27a-9314360604dd/volumes"
Oct 31 19:39:22.879374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.879339    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:22.882521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:22.882485    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:23.106079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:23.104870    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:23.772037 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:39:23.772008    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d76eb2a_ecc5_4b45_a134_754216e355c0.slice/crio-24f34050d935ad87ba3bdf1b12d018f106bfdce8c5b6ce27902cfc3bd270d580 WatchSource:0}: Error finding container 24f34050d935ad87ba3bdf1b12d018f106bfdce8c5b6ce27902cfc3bd270d580: Status 404 returned error can't find the container with id 24f34050d935ad87ba3bdf1b12d018f106bfdce8c5b6ce27902cfc3bd270d580
Oct 31 19:39:23.774483 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:23.774454    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"]
Oct 31 19:39:24.489539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:24.489496    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" event={"ID":"0d76eb2a-ecc5-4b45-a134-754216e355c0","Type":"ContainerStarted","Data":"7b458b6d03cc70bd6bc6265034af816c7936f526699c0a5cad774c14f1a0b74f"}
Oct 31 19:39:24.489539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:24.489579    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" event={"ID":"0d76eb2a-ecc5-4b45-a134-754216e355c0","Type":"ContainerStarted","Data":"24f34050d935ad87ba3bdf1b12d018f106bfdce8c5b6ce27902cfc3bd270d580"}
Oct 31 19:39:24.490059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:24.489610    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:24.515397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:24.515290    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" podStartSLOduration=2.023260766 podStartE2EDuration="2.515276309s" totalImagesPullingTime="492.015543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:24.512880794 +0000 UTC m=+1157.974255426" watchObservedRunningTime="2026-10-31 19:39:24.515276309 +0000 UTC m=+1157.976650965"
Oct 31 19:39:25.040329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:25.040296    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"]
Oct 31 19:39:25.044669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:25.044640    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5035d3f3-pqkd9"]
Oct 31 19:39:26.666891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:26.666854    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4962e0-e4d0-45de-8909-4b28c3777159" path="/var/lib/kubelet/pods/ad4962e0-e4d0-45de-8909-4b28c3777159/volumes"
Oct 31 19:39:27.257682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:27.257654    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:39:30.514566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:30.508421    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 19:39:31.566565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:31.566520    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:39:33.051152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:33.051090    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"]
Oct 31 19:39:33.151389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:33.151353    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc56b863-m22km"]
Oct 31 19:39:34.901012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:34.900791    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d51830-2bad-4364-b14b-ed7bf57c9a31" path="/var/lib/kubelet/pods/e5d51830-2bad-4364-b14b-ed7bf57c9a31/volumes"
Oct 31 19:39:47.073661 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:39:47.073584    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:39:54.395309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.395271    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"]
Oct 31 19:39:54.400720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.400696    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:54.406103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.406075    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e73256\""
Oct 31 19:39:54.409154 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.409126    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"]
Oct 31 19:39:54.422455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.422424    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e73256-dockercfg-jd5zc\""
Oct 31 19:39:54.584572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.550633    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:54.584572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.551063    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx\") pod \"payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:54.678624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.653348    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx\") pod \"payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:54.678624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.653572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:54.678624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.654907    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:54.906007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:54.905944    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx\") pod \"payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:55.036154 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:55.036061    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:55.370607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:55.370021    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"]
Oct 31 19:39:55.373216 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:39:55.372841    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef822f74_c7b1_4f26_bc53_4b73a9991643.slice/crio-7c0d3a0648b10d33aa7076c18ef66251bf88210b0beda5557099f12696a8261c WatchSource:0}: Error finding container 7c0d3a0648b10d33aa7076c18ef66251bf88210b0beda5557099f12696a8261c: Status 404 returned error can't find the container with id 7c0d3a0648b10d33aa7076c18ef66251bf88210b0beda5557099f12696a8261c
Oct 31 19:39:56.359706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:56.359669    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9" event={"ID":"ef822f74-c7b1-4f26-bc53-4b73a9991643","Type":"ContainerStarted","Data":"443eec9a7947681501c564f5e6d32783c8e560565f6e8f8ce6ac5422dc5e1ab2"}
Oct 31 19:39:56.360116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:56.359722    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9" event={"ID":"ef822f74-c7b1-4f26-bc53-4b73a9991643","Type":"ContainerStarted","Data":"7c0d3a0648b10d33aa7076c18ef66251bf88210b0beda5557099f12696a8261c"}
Oct 31 19:39:56.360116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:56.359889    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:39:56.383042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:56.382986    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9" podStartSLOduration=2.382971121 podStartE2EDuration="2.382971121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:56.380429604 +0000 UTC m=+1189.841804220" watchObservedRunningTime="2026-10-31 19:39:56.382971121 +0000 UTC m=+1189.844345735"
Oct 31 19:39:56.888187 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:56.888150    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"]
Oct 31 19:39:56.909476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:39:56.909440    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"]
Oct 31 19:40:01.888642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:01.888593    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62" podUID="0410f536-f2eb-42bc-801f-c8a744546ebf" containerName="payload-processing" containerID="cri-o://f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710" gracePeriod=25
Oct 31 19:40:01.909828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:01.909772    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx" podUID="ceba3e96-a367-438c-816d-8e119418a22a" containerName="payload-pre-processing" containerID="cri-o://dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184" gracePeriod=25
Oct 31 19:40:02.249649 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.249622    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:40:02.295278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.295237    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54\" (UniqueName: \"kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54\") pod \"0410f536-f2eb-42bc-801f-c8a744546ebf\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") "
Oct 31 19:40:02.295710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.295683    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume\") pod \"0410f536-f2eb-42bc-801f-c8a744546ebf\" (UID: \"0410f536-f2eb-42bc-801f-c8a744546ebf\") "
Oct 31 19:40:02.296156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.296127    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume" pod "0410f536-f2eb-42bc-801f-c8a744546ebf" (UID: "0410f536-f2eb-42bc-801f-c8a744546ebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:02.299152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.299117    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54" pod "0410f536-f2eb-42bc-801f-c8a744546ebf" (UID: "0410f536-f2eb-42bc-801f-c8a744546ebf"). InnerVolumeSpecName "kube-api-access-d9k54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:02.303177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.303146    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0410f536-f2eb-42bc-801f-c8a744546ebf-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:02.303177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.303172    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:40:02.303372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.303181    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9k54\" (UniqueName: \"kubernetes.io/projected/0410f536-f2eb-42bc-801f-c8a744546ebf-kube-api-access-d9k54\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:02.404898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.404864    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume\") pod \"ceba3e96-a367-438c-816d-8e119418a22a\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") "
Oct 31 19:40:02.405073 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.405059    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr\" (UniqueName: \"kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr\") pod \"ceba3e96-a367-438c-816d-8e119418a22a\" (UID: \"ceba3e96-a367-438c-816d-8e119418a22a\") "
Oct 31 19:40:02.405327 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.405305    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume" pod "ceba3e96-a367-438c-816d-8e119418a22a" (UID: "ceba3e96-a367-438c-816d-8e119418a22a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:02.405786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.405756    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceba3e96-a367-438c-816d-8e119418a22a-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:02.407613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.407411    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr" pod "ceba3e96-a367-438c-816d-8e119418a22a" (UID: "ceba3e96-a367-438c-816d-8e119418a22a"). InnerVolumeSpecName "kube-api-access-5fstr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:02.524386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.524285    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fstr\" (UniqueName: \"kubernetes.io/projected/ceba3e96-a367-438c-816d-8e119418a22a-kube-api-access-5fstr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:02.589497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.589458    5958 generic.go:356] "Generic (PLEG): container finished" podID="0410f536-f2eb-42bc-801f-c8a744546ebf" containerID="f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710" exitCode=0
Oct 31 19:40:02.589794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.589765    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62" event={"ID":"0410f536-f2eb-42bc-801f-c8a744546ebf","Type":"ContainerDied","Data":"f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710"}
Oct 31 19:40:02.589906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.589859    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62" event={"ID":"0410f536-f2eb-42bc-801f-c8a744546ebf","Type":"ContainerDied","Data":"d954daa15927f0979cbb0aae0613e863bd288b3784d1d877fff657092a98a50a"}
Oct 31 19:40:02.589972 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.589918    5958 scope.go:122] "RemoveContainer" containerID="f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710"
Oct 31 19:40:02.590511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.590211    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"
Oct 31 19:40:02.592041 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.591879    5958 generic.go:356] "Generic (PLEG): container finished" podID="ceba3e96-a367-438c-816d-8e119418a22a" containerID="dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184" exitCode=0
Oct 31 19:40:02.592041 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.592003    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx" event={"ID":"ceba3e96-a367-438c-816d-8e119418a22a","Type":"ContainerDied","Data":"dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184"}
Oct 31 19:40:02.592179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.592043    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx" event={"ID":"ceba3e96-a367-438c-816d-8e119418a22a","Type":"ContainerDied","Data":"963fc78e5b06b0be282102c8b1535b15dcbe9849631390b961141d6762dad4d3"}
Oct 31 19:40:02.592179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.592109    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"
Oct 31 19:40:02.617449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.617426    5958 scope.go:122] "RemoveContainer" containerID="f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710"
Oct 31 19:40:02.623303 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:02.619445    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710\": container with ID starting with f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710 not found: ID does not exist" containerID="f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710"
Oct 31 19:40:02.623303 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.619488    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710"} err="failed to get container status \"f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710\": rpc error: code = NotFound desc = could not find container \"f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710\": container with ID starting with f151a0a8ef1a60a1cc74ff5466af8e3fb1edaa3c4735f0306efe92dd3d378710 not found: ID does not exist"
Oct 31 19:40:02.623303 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.619512    5958 scope.go:122] "RemoveContainer" containerID="dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184"
Oct 31 19:40:02.637858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.637832    5958 scope.go:122] "RemoveContainer" containerID="dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184"
Oct 31 19:40:02.638266 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:02.638239    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184\": container with ID starting with dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184 not found: ID does not exist" containerID="dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184"
Oct 31 19:40:02.638398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.638277    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184"} err="failed to get container status \"dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184\": rpc error: code = NotFound desc = could not find container \"dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184\": container with ID starting with dd4419a8d460491dd0d96ebad65ff9607c3cbfba825d806c9fb6908fdbf4d184 not found: ID does not exist"
Oct 31 19:40:02.641368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.641340    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"]
Oct 31 19:40:02.647365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.647340    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9e323f-778fbb59d7-49m62"]
Oct 31 19:40:02.681939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.681889    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0410f536-f2eb-42bc-801f-c8a744546ebf" path="/var/lib/kubelet/pods/0410f536-f2eb-42bc-801f-c8a744546ebf/volumes"
Oct 31 19:40:02.682763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.682737    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"]
Oct 31 19:40:02.701149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:02.701114    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9e323f-85744fc5f9-kjwgx"]
Oct 31 19:40:05.105612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:05.105573    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceba3e96-a367-438c-816d-8e119418a22a" path="/var/lib/kubelet/pods/ceba3e96-a367-438c-816d-8e119418a22a/volumes"
Oct 31 19:40:05.107065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:05.107042    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"]
Oct 31 19:40:05.107512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:05.107486    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" containerID="cri-o://4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e" gracePeriod=30
Oct 31 19:40:06.910242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:06.905791    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:40:06.910242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:06.905865    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:40:07.466745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:07.461748    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:40:07.963531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:07.963500    5958 scope.go:122] "RemoveContainer" containerID="225ccfd3752ce9ce95de152d7eadaab8e38f64a772e28d8f158d988cd6937e2a"
Oct 31 19:40:07.974750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:07.974724    5958 scope.go:122] "RemoveContainer" containerID="b2b7c0ec29b03e7dd5f5250feed139d0dbda294266cdfb16b7f26605f7a19d5f"
Oct 31 19:40:07.989037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:07.989011    5958 scope.go:122] "RemoveContainer" containerID="c8da6ebf76d968f33223efd1b826978f032778c6fdf4daa30ec62345d9794065"
Oct 31 19:40:08.002509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:08.002483    5958 scope.go:122] "RemoveContainer" containerID="b803c184b1cf153d941346d1bd9eaddae151ee1950f8e9ebc0853fbd3694421a"
Oct 31 19:40:08.014007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:08.013984    5958 scope.go:122] "RemoveContainer" containerID="db955bfeafde91029ba8af4883825b803264f50c7bfd5ec2110ea3dc4c93a3b0"
Oct 31 19:40:10.861530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.861494    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:40:10.895832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.895799    5958 generic.go:356] "Generic (PLEG): container finished" podID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerID="4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e" exitCode=0
Oct 31 19:40:10.896308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.895835    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"
Oct 31 19:40:10.896483 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.895861    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" event={"ID":"1655e665-7376-41cf-b2ab-44a2d1405bad","Type":"ContainerDied","Data":"4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e"}
Oct 31 19:40:10.896632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.896480    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm" event={"ID":"1655e665-7376-41cf-b2ab-44a2d1405bad","Type":"ContainerDied","Data":"133df5c14cf6f94679ab442759aabbb5a15600204da491c16d4fa6fc7eabdce1"}
Oct 31 19:40:10.896632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.896518    5958 scope.go:122] "RemoveContainer" containerID="4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e"
Oct 31 19:40:10.920784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.919139    5958 scope.go:122] "RemoveContainer" containerID="4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e"
Oct 31 19:40:10.920784 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:10.919586    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e\": container with ID starting with 4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e not found: ID does not exist" containerID="4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e"
Oct 31 19:40:10.920784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.919626    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e"} err="failed to get container status \"4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e\": rpc error: code = NotFound desc = could not find container \"4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e\": container with ID starting with 4c23dad2bb7cba9593d93773d1624640752be8cebbe8f0e1b8f9d84db267d86e not found: ID does not exist"
Oct 31 19:40:10.926435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.926397    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.926827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.926801    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.926947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.926840    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.926947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.926902    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:10.927033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927013    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.927074 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927057    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.927121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927105    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.927190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927175    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.927253 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927235    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.927378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927361    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy\") pod \"1655e665-7376-41cf-b2ab-44a2d1405bad\" (UID: \"1655e665-7376-41cf-b2ab-44a2d1405bad\") "
Oct 31 19:40:10.927529 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927503    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:10.927709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927685    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:10.927910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.927888    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:10.928355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.928284    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:10.928355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.928304    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1655e665-7376-41cf-b2ab-44a2d1405bad-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:10.928355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.928313    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:10.928355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.928323    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:10.930379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.929676    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "kube-api-access-g45sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:10.930379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.929987    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:10.930720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.930696    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:10.932036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.932002    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:10.936711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:10.936676    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy" pod "1655e665-7376-41cf-b2ab-44a2d1405bad" (UID: "1655e665-7376-41cf-b2ab-44a2d1405bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:11.030283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.030246    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:11.030283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.030280    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:11.030283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.030290    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:11.030522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.030298    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:11.030522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.030310    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45sw\" (UniqueName: \"kubernetes.io/projected/1655e665-7376-41cf-b2ab-44a2d1405bad-kube-api-access-g45sw\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:11.227657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.227604    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"]
Oct 31 19:40:11.232091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:11.232059    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ckfzm"]
Oct 31 19:40:12.670244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.670211    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" path="/var/lib/kubelet/pods/1655e665-7376-41cf-b2ab-44a2d1405bad/volumes"
Oct 31 19:40:12.863177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.863141    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"]
Oct 31 19:40:12.864365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864338    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0410f536-f2eb-42bc-801f-c8a744546ebf" containerName="payload-processing"
Oct 31 19:40:12.864365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864361    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0410f536-f2eb-42bc-801f-c8a744546ebf" containerName="payload-processing"
Oct 31 19:40:12.864578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864391    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy"
Oct 31 19:40:12.864578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864397    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy"
Oct 31 19:40:12.864578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864408    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ceba3e96-a367-438c-816d-8e119418a22a" containerName="payload-pre-processing"
Oct 31 19:40:12.864578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864416    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceba3e96-a367-438c-816d-8e119418a22a" containerName="payload-pre-processing"
Oct 31 19:40:12.864707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864612    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0410f536-f2eb-42bc-801f-c8a744546ebf" containerName="payload-processing"
Oct 31 19:40:12.864707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864635    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1655e665-7376-41cf-b2ab-44a2d1405bad" containerName="istio-proxy"
Oct 31 19:40:12.864707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.864643    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceba3e96-a367-438c-816d-8e119418a22a" containerName="payload-pre-processing"
Oct 31 19:40:12.871337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.871305    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:12.881927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.881890    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"]
Oct 31 19:40:12.884655 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.882367    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-03dfee\""
Oct 31 19:40:12.885128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.885107    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-03dfee-dockercfg-cnhn4\""
Oct 31 19:40:12.892723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.892700    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"]
Oct 31 19:40:12.899988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.899956    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:12.913719 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:12.913687    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"]
Oct 31 19:40:13.106618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.102032    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtzt\" (UniqueName: \"kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt\") pod \"payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.106618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.102235    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.106618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.102373    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:13.106618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.102426    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xks5z\" (UniqueName: \"kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z\") pod \"payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:13.229566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.216236    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.229566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.216385    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:13.229566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.216460    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xks5z\" (UniqueName: \"kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z\") pod \"payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:13.229566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.216941    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtzt\" (UniqueName: \"kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt\") pod \"payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.229566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.218614    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.229566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.219401    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:13.259317 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.259284    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xks5z\" (UniqueName: \"kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z\") pod \"payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:13.275717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.262310    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtzt\" (UniqueName: \"kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt\") pod \"payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.523610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.523033    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:13.527656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:13.526135    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:14.114670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:14.103796    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"]
Oct 31 19:40:14.114670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:14.110702    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"]
Oct 31 19:40:15.026472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.026432    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" event={"ID":"280ebe12-7060-4a50-885a-c20765df1460","Type":"ContainerStarted","Data":"1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01"}
Oct 31 19:40:15.026702 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.026503    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:15.026702 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.026530    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" event={"ID":"280ebe12-7060-4a50-885a-c20765df1460","Type":"ContainerStarted","Data":"f3553d936ac2a417101202bcc0b606183ead5b3e88902999eee4d6e165c5b573"}
Oct 31 19:40:15.028281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.028251    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" event={"ID":"65bf01c1-da58-4383-a303-8ffc1ea1289b","Type":"ContainerStarted","Data":"cc62417c7dd942c7130724d7688613bfa7ff2b86c1ab4b64338f3a43b4b1f043"}
Oct 31 19:40:15.028422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.028309    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" event={"ID":"65bf01c1-da58-4383-a303-8ffc1ea1289b","Type":"ContainerStarted","Data":"3bb663e90e7b252d19221d03101db98c50d8ecb455cf8e99dadbf988d4862d99"}
Oct 31 19:40:15.028422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.028341    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:15.160444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.160394    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" podStartSLOduration=3.160380519 podStartE2EDuration="3.160380519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:15.157841855 +0000 UTC m=+1208.619216470" watchObservedRunningTime="2026-10-31 19:40:15.160380519 +0000 UTC m=+1208.621755134"
Oct 31 19:40:15.161071 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:15.161040    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" podStartSLOduration=3.161033571 podStartE2EDuration="3.161033571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:15.127116762 +0000 UTC m=+1208.588491376" watchObservedRunningTime="2026-10-31 19:40:15.161033571 +0000 UTC m=+1208.622408186"
Oct 31 19:40:17.048287 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:17.048251    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:40:18.756444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:18.756407    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:40:20.782161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.782112    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"]
Oct 31 19:40:20.813923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.813885    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"]
Oct 31 19:40:20.814090 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.814022    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.821591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.821337    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-fd1c91-serving-cert\""
Oct 31 19:40:20.871259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.871217    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.871651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.871626    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.871771 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.871733    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.871837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.871824    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cpf6\" (UniqueName: \"kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.976320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.976274    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.980981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.980939    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.981407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.981374    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cpf6\" (UniqueName: \"kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.981627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.981587    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:20.983000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:20.980722    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:21.043663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:21.037070    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:21.043663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:21.043129    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cpf6\" (UniqueName: \"kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:21.059597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:21.058320    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:21.134220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:21.134174    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:21.428466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:21.428423    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"]
Oct 31 19:40:21.430535 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:40:21.430485    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e51ec0_aac2_4216_8f04_7f0fc1bfd9d9.slice/crio-e909d9630964418f08b48819df683f4843e8ae48ff415ce48b6ef2b46fe5274c WatchSource:0}: Error finding container e909d9630964418f08b48819df683f4843e8ae48ff415ce48b6ef2b46fe5274c: Status 404 returned error can't find the container with id e909d9630964418f08b48819df683f4843e8ae48ff415ce48b6ef2b46fe5274c
Oct 31 19:40:21.432502 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:21.432475    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:40:22.245666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:22.245572    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96" event={"ID":"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9","Type":"ContainerStarted","Data":"fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126"}
Oct 31 19:40:22.245666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:22.245623    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96" event={"ID":"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9","Type":"ContainerStarted","Data":"e909d9630964418f08b48819df683f4843e8ae48ff415ce48b6ef2b46fe5274c"}
Oct 31 19:40:22.246108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:22.245807    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:22.271599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:22.271531    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96" podStartSLOduration=1.733855661 podStartE2EDuration="2.271516938s" totalImagesPullingTime="537.661277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:22.26804061 +0000 UTC m=+1215.729415225" watchObservedRunningTime="2026-10-31 19:40:22.271516938 +0000 UTC m=+1215.732891585"
Oct 31 19:40:24.395738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:24.393602    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"]
Oct 31 19:40:24.395738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:24.393837    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" podUID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" containerName="maas-api" containerID="cri-o://a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74" gracePeriod=30
Oct 31 19:40:25.194567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.192317    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:40:25.303573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.298997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk\" (UniqueName: \"kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk\") pod \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") "
Oct 31 19:40:25.303573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.299691    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs\") pod \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") "
Oct 31 19:40:25.303573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.300295    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls\") pod \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") "
Oct 31 19:40:25.303573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.300346    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca\") pod \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\" (UID: \"0c477b14-5390-42fd-a54b-5a5ba5e6fa27\") "
Oct 31 19:40:25.303573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.301801    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca" pod "0c477b14-5390-42fd-a54b-5a5ba5e6fa27" (UID: "0c477b14-5390-42fd-a54b-5a5ba5e6fa27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:25.303573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.303388    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs" pod "0c477b14-5390-42fd-a54b-5a5ba5e6fa27" (UID: "0c477b14-5390-42fd-a54b-5a5ba5e6fa27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:25.304016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.303995    5958 generic.go:356] "Generic (PLEG): container finished" podID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" containerID="a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74" exitCode=0
Oct 31 19:40:25.307695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.304122    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" event={"ID":"0c477b14-5390-42fd-a54b-5a5ba5e6fa27","Type":"ContainerDied","Data":"a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74"}
Oct 31 19:40:25.307695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.304187    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" event={"ID":"0c477b14-5390-42fd-a54b-5a5ba5e6fa27","Type":"ContainerDied","Data":"a9688cb87b7fa764037fbe3ff1f5fbb3551339c519dd9b58b1000d546d2e8566"}
Oct 31 19:40:25.307695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.304351    5958 scope.go:122] "RemoveContainer" containerID="a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74"
Oct 31 19:40:25.307695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.304592    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"
Oct 31 19:40:25.311599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.310005    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk" pod "0c477b14-5390-42fd-a54b-5a5ba5e6fa27" (UID: "0c477b14-5390-42fd-a54b-5a5ba5e6fa27"). InnerVolumeSpecName "kube-api-access-qjttk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:25.315345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.315313    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls" pod "0c477b14-5390-42fd-a54b-5a5ba5e6fa27" (UID: "0c477b14-5390-42fd-a54b-5a5ba5e6fa27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:25.358164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.358132    5958 scope.go:122] "RemoveContainer" containerID="a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74"
Oct 31 19:40:25.359595 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:25.358684    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74\": container with ID starting with a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74 not found: ID does not exist" containerID="a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74"
Oct 31 19:40:25.359595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.358728    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74"} err="failed to get container status \"a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74\": rpc error: code = NotFound desc = could not find container \"a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74\": container with ID starting with a4903b4caac09ee2f27a57c64d234ad5eb698b2bd905bc15a4aaf1049a792c74 not found: ID does not exist"
Oct 31 19:40:25.403079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.403040    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:25.403079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.403081    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:25.403501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.403099    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:25.403501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.403114    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjttk\" (UniqueName: \"kubernetes.io/projected/0c477b14-5390-42fd-a54b-5a5ba5e6fa27-kube-api-access-qjttk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:25.640067 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.640031    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"]
Oct 31 19:40:25.645816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:25.645783    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2"]
Oct 31 19:40:25.698687 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:25.698652    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c477b14_5390_42fd_a54b_5a5ba5e6fa27.slice/crio-a9688cb87b7fa764037fbe3ff1f5fbb3551339c519dd9b58b1000d546d2e8566\": RecentStats: unable to find data in memory cache]"
Oct 31 19:40:26.042474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:26.042437    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:40:26.042862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:26.042845    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:40:26.668704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:26.668661    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" path="/var/lib/kubelet/pods/0c477b14-5390-42fd-a54b-5a5ba5e6fa27/volumes"
Oct 31 19:40:28.123201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:28.123151    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9e323f-65ffc8f7c9-fwcg2" podUID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:28.500589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:28.491157    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.099693    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"]
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.110144    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" containerName="maas-api"
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.110168    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" containerName="maas-api"
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.110569    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c477b14-5390-42fd-a54b-5a5ba5e6fa27" containerName="maas-api"
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.128516    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"]
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.128649    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.137568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.134901    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d4b21c-openshift-default-dockercfg-k8zbt\""
Oct 31 19:40:30.294348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294311    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294360    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294434    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294526    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294676    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294718    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294808    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294860    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.294919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.294896    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8hz\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.398622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.398388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.398622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.398484    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.398622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.398562    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.398931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.398629    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.398931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.398680    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8hz\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.404609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.399211    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.404609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.399269    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.404609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.399301    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.404609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.399419    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.404609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.399993    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.404609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.400829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.420687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.420651    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.421429 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.421397    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.421722 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.421488    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.447981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.447935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.452195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.452155    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.453037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.453007    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8hz\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.483461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.483413    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:30.750117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:30.750026    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:31.228568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:31.224108    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"]
Oct 31 19:40:31.249458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:31.249007    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:31.249458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:31.249163    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:31.249458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:31.249248    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:31.265181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:31.265131    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" event={"ID":"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262","Type":"ContainerStarted","Data":"e39bfa83b7996bf54bcbd3792769bcaf3af021a4f6275a46740ef5dc0893e53e"}
Oct 31 19:40:32.275326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.275287    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" event={"ID":"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262","Type":"ContainerStarted","Data":"b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8"}
Oct 31 19:40:32.335200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.335109    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podStartSLOduration=2.335088432 podStartE2EDuration="2.335088432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:32.324404655 +0000 UTC m=+1225.785779272" watchObservedRunningTime="2026-10-31 19:40:32.335088432 +0000 UTC m=+1225.796463049"
Oct 31 19:40:32.688019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.687987    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"]
Oct 31 19:40:32.695502 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.695473    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.699938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.699904    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"]
Oct 31 19:40:32.701362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.701336    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67adab47-openshift-default-dockercfg-bbdfj\""
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750124    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750227    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750299    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750362    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750417    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750571    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750776    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.750997    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.751022    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.752592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.751343    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:32.758384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.755494    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:32.758384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.755563    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.859526    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.859619    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.859759    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.859944    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.860028    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.860055    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.860142    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.860206    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.860283    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.861241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.860851    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.862715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.862421    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.901496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.893362    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.901496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.896456    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.914579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.912378    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.915792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.915761    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.935638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.930046    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.956661 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.956585    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:32.957469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:32.957440    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:33.063558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:33.063515    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:33.432814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:33.430767    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"]
Oct 31 19:40:33.433960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:33.433925    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:33.434084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:33.433996    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:33.434084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:33.434027    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:34.052603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.046692    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:34.052603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.046778    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:34.474308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.472800    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" event={"ID":"ae6edde9-79e2-4fb9-ac86-d0c7626c615a","Type":"ContainerStarted","Data":"ab59594e92d15e545d00edc80d8ad96def129745b599d645c7889cdd8f5d05da"}
Oct 31 19:40:34.474308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.472880    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" event={"ID":"ae6edde9-79e2-4fb9-ac86-d0c7626c615a","Type":"ContainerStarted","Data":"9501ecffef07712c9adf4ddbf9bd38dc396af19da7f3d6f293676901b72468c1"}
Oct 31 19:40:34.553996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.550178    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podStartSLOduration=2.550155056 podStartE2EDuration="2.550155056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:34.544303449 +0000 UTC m=+1228.005678065" watchObservedRunningTime="2026-10-31 19:40:34.550155056 +0000 UTC m=+1228.011529672"
Oct 31 19:40:34.751497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.751402    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:34.751665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:34.751496    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:35.080684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:35.067358    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:35.080684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:35.067483    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:35.080684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:35.067525    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:35.758986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:35.758946    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:35.759625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:35.759590    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:36.064618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:36.063998    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:36.064618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:36.064064    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:36.185911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:36.183766    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"]
Oct 31 19:40:36.281538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:36.276145    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"]
Oct 31 19:40:36.756572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:36.752568    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:36.756572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:36.752627    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:37.070684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:37.070111    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:37.070684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:37.070178    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:37.759654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:37.759155    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:37.759654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:37.759223    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:38.064045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:38.063945    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:38.064045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:38.064005    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:38.864565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:38.859974    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 19:40:38.864565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:38.860040    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 19:40:39.064035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:39.063997    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:39.064244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:39.064062    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:39.927087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:39.927048    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:40.058563 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:40.057200    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:40.073908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:40.073872    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:40.087566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:40.082064    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:40.087566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:40.076374    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:40:41.064001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.063968    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:41.064416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.064024    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:41.184633 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.184570    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7" podUID="2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" containerName="payload-processing" containerID="cri-o://fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4" gracePeriod=25
Oct 31 19:40:41.277593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.277493    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr" podUID="ab4f944d-327a-4e39-82c3-79a90a0d7077" containerName="payload-pre-processing" containerID="cri-o://cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d" gracePeriod=25
Oct 31 19:40:41.868617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.868583    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:40:41.952215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.951935    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume\") pod \"ab4f944d-327a-4e39-82c3-79a90a0d7077\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") "
Oct 31 19:40:41.952215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.952104    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx\" (UniqueName: \"kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx\") pod \"ab4f944d-327a-4e39-82c3-79a90a0d7077\" (UID: \"ab4f944d-327a-4e39-82c3-79a90a0d7077\") "
Oct 31 19:40:41.952693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.952640    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume" pod "ab4f944d-327a-4e39-82c3-79a90a0d7077" (UID: "ab4f944d-327a-4e39-82c3-79a90a0d7077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:41.953313 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.953239    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4f944d-327a-4e39-82c3-79a90a0d7077-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.955778 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.955720    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx" pod "ab4f944d-327a-4e39-82c3-79a90a0d7077" (UID: "ab4f944d-327a-4e39-82c3-79a90a0d7077"). InnerVolumeSpecName "kube-api-access-xsjpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.992338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:41.992309    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:40:42.057078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.057039    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsjpx\" (UniqueName: \"kubernetes.io/projected/ab4f944d-327a-4e39-82c3-79a90a0d7077-kube-api-access-xsjpx\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:42.065255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.065159    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 19:40:42.065255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.065212    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 19:40:42.155797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.155760    5958 generic.go:356] "Generic (PLEG): container finished" podID="2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" containerID="fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4" exitCode=0
Oct 31 19:40:42.155797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.155803    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"
Oct 31 19:40:42.156077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.155845    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7" event={"ID":"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c","Type":"ContainerDied","Data":"fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4"}
Oct 31 19:40:42.156077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.155895    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7" event={"ID":"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c","Type":"ContainerDied","Data":"96121a1c59785a1937f414a3f23eb15a29cfcb085c123c2aa3078d567df0fe89"}
Oct 31 19:40:42.156077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.155919    5958 scope.go:122] "RemoveContainer" containerID="fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4"
Oct 31 19:40:42.157695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.157669    5958 generic.go:356] "Generic (PLEG): container finished" podID="ab4f944d-327a-4e39-82c3-79a90a0d7077" containerID="cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d" exitCode=0
Oct 31 19:40:42.157847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.157713    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume\") pod \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") "
Oct 31 19:40:42.157847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.157742    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"
Oct 31 19:40:42.157847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.157754    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr" event={"ID":"ab4f944d-327a-4e39-82c3-79a90a0d7077","Type":"ContainerDied","Data":"cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d"}
Oct 31 19:40:42.157847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.157798    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr" event={"ID":"ab4f944d-327a-4e39-82c3-79a90a0d7077","Type":"ContainerDied","Data":"3489aa1412e2e7c7de9df2e9704efa6688c2e2407f10af8ba98be0358466f5ca"}
Oct 31 19:40:42.158094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.157888    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr\" (UniqueName: \"kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr\") pod \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\" (UID: \"2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c\") "
Oct 31 19:40:42.158279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.158248    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume" pod "2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" (UID: "2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:42.158838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.158821    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:42.160488 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.160460    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr" pod "2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" (UID: "2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c"). InnerVolumeSpecName "kube-api-access-tmrzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:42.169863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.169838    5958 scope.go:122] "RemoveContainer" containerID="fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4"
Oct 31 19:40:42.170186 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:42.170159    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4\": container with ID starting with fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4 not found: ID does not exist" containerID="fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4"
Oct 31 19:40:42.170269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.170199    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4"} err="failed to get container status \"fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4\": rpc error: code = NotFound desc = could not find container \"fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4\": container with ID starting with fb415ab4939991a2d0e0d21759dfdf2bc2406f917f186d9616f5eaddc2a703c4 not found: ID does not exist"
Oct 31 19:40:42.170269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.170232    5958 scope.go:122] "RemoveContainer" containerID="cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d"
Oct 31 19:40:42.182238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.182217    5958 scope.go:122] "RemoveContainer" containerID="cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d"
Oct 31 19:40:42.182811 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:42.182740    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d\": container with ID starting with cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d not found: ID does not exist" containerID="cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d"
Oct 31 19:40:42.182978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.182821    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d"} err="failed to get container status \"cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d\": rpc error: code = NotFound desc = could not find container \"cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d\": container with ID starting with cc267ca575258d7a0d4baf26fc8618f4b412ee9f4dadd98337d3f2bbdc36e86d not found: ID does not exist"
Oct 31 19:40:42.189440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.189410    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"]
Oct 31 19:40:42.193833 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.193809    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-db61066b-7fd4c8cb6c-cf2lr"]
Oct 31 19:40:42.260712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.260672    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrzr\" (UniqueName: \"kubernetes.io/projected/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c-kube-api-access-tmrzr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:42.491332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.491302    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"]
Oct 31 19:40:42.498148 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.498116    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db61066b-5cbb466f49-5zjw7"]
Oct 31 19:40:42.751645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.751279    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" path="/var/lib/kubelet/pods/2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c/volumes"
Oct 31 19:40:42.781262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:42.776978    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4f944d-327a-4e39-82c3-79a90a0d7077" path="/var/lib/kubelet/pods/ab4f944d-327a-4e39-82c3-79a90a0d7077/volumes"
Oct 31 19:40:43.069597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:43.069500    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:43.070008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:43.069993    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:43.070738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:43.070713    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:40:47.051447 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:40:47.051380    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:40:47.736265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.736233    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"]
Oct 31 19:40:47.737514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.737487    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" containerName="payload-processing"
Oct 31 19:40:47.737514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.737513    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" containerName="payload-processing"
Oct 31 19:40:47.737760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.737587    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ab4f944d-327a-4e39-82c3-79a90a0d7077" containerName="payload-pre-processing"
Oct 31 19:40:47.737760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.737598    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4f944d-327a-4e39-82c3-79a90a0d7077" containerName="payload-pre-processing"
Oct 31 19:40:47.737843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.737834    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e1497f0-b3ab-4453-a5e8-7d83b3e75f6c" containerName="payload-processing"
Oct 31 19:40:47.737878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.737857    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4f944d-327a-4e39-82c3-79a90a0d7077" containerName="payload-pre-processing"
Oct 31 19:40:47.745221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.745153    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.753534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.753504    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"]
Oct 31 19:40:47.874251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874195    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874336    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874392    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874432    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874600    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874653    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874812    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874844    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.874906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.874879    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5q9\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976475    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976527    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976607    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5q9\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976674    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976756    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976819    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.976990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976878    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.976992    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.977016    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.977059    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.977227    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.977363    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.977538    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.977719 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.977680    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.979175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.979154    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.979262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.979250    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.990029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.989964    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:47.990197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:47.990176    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5q9\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:48.084039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:48.075523    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:48.462425 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:40:48.462388    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933a8af3_fa2c_4bd9_a51c_9d2295f28c19.slice/crio-e54cb31c5ea6179590a1d443e557a50bba8458c9ad347d0a59f31f43468b38ff WatchSource:0}: Error finding container e54cb31c5ea6179590a1d443e557a50bba8458c9ad347d0a59f31f43468b38ff: Status 404 returned error can't find the container with id e54cb31c5ea6179590a1d443e557a50bba8458c9ad347d0a59f31f43468b38ff
Oct 31 19:40:48.463918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:48.463889    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"]
Oct 31 19:40:48.465673 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:48.465638    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:48.465757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:48.465707    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:48.465757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:48.465738    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:40:49.279343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:49.279303    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" event={"ID":"933a8af3-fa2c-4bd9-a51c-9d2295f28c19","Type":"ContainerStarted","Data":"7b41f43e1c9d3a09eb11442f0d1404ae721163385d8afa15872d0b067170b161"}
Oct 31 19:40:49.279772 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:49.279362    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" event={"ID":"933a8af3-fa2c-4bd9-a51c-9d2295f28c19","Type":"ContainerStarted","Data":"e54cb31c5ea6179590a1d443e557a50bba8458c9ad347d0a59f31f43468b38ff"}
Oct 31 19:40:49.319539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:49.319486    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podStartSLOduration=2.319470754 podStartE2EDuration="2.319470754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:49.315117383 +0000 UTC m=+1242.776492000" watchObservedRunningTime="2026-10-31 19:40:49.319470754 +0000 UTC m=+1242.780845368"
Oct 31 19:40:50.109567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:50.105797    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:50.109567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:50.106298    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:50.109567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:50.106363    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:51.076127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:51.076089    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:51.076530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:51.076162    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:52.081569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:52.081206    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:52.081569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:52.081274    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:53.130054 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:53.130004    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:53.130675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:53.130628    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:54.081268 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:54.081231    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:54.081694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:54.081646    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:55.076505 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:55.076469    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:55.076948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:55.076618    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:56.352568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:56.349172    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 31 19:40:56.352568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:56.349262    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 31 19:40:57.684568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:57.676846    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:57.686870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:57.686843    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:40:57.691927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:40:57.690631    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:41:02.796586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.796533    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"]
Oct 31 19:41:02.824526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.809936    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.872847    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873047    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873161    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873226    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873288    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873409    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p58k\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873587    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873638    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.874114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.873702    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.912183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.912148    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"]
Oct 31 19:41:02.974904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.974871    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.974934    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975021    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p58k\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975120    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975138    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975181    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975248    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975328    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975356    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975366    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975535    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.975852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.975829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.976026 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.976002    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.976177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.976159    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.977700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.977682    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.977874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.977853    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.989040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.989008    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p58k\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:02.989402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:02.989380    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:03.215236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:03.214943    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:03.994220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:03.994171    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"]
Oct 31 19:41:04.024577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:04.022328    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:04.024577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:04.022443    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:04.024577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:04.024068    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:05.047859 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:05.047810    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" event={"ID":"71d3474d-ff6c-488e-9c18-0ded72b45f30","Type":"ContainerStarted","Data":"f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa"}
Oct 31 19:41:05.048323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:05.047873    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" event={"ID":"71d3474d-ff6c-488e-9c18-0ded72b45f30","Type":"ContainerStarted","Data":"3028e5f2ceddd46061629bc6bff68219b990b49775bfef7e162247284e4fd759"}
Oct 31 19:41:05.087970 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:05.087905    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podStartSLOduration=3.087885178 podStartE2EDuration="3.087885178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:05.081120408 +0000 UTC m=+1258.542495026" watchObservedRunningTime="2026-10-31 19:41:05.087885178 +0000 UTC m=+1258.549259792"
Oct 31 19:41:05.230724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:05.224837    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:05.230724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:05.226993    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:05.230724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:05.227052    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:06.228691 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:06.228650    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:06.229318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:06.229258    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:07.223988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:07.223949    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:07.229133 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:07.228699    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:08.089732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:08.089699    5958 scope.go:122] "RemoveContainer" containerID="2058747992dc900a47d29972b6bf314438634b1a8478af16e18f003bd0790eb4"
Oct 31 19:41:08.101391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:08.101364    5958 scope.go:122] "RemoveContainer" containerID="b6a55ebe0df9a5b12ebfe888544fc646881faba43f9289c3ec4d10134a83109a"
Oct 31 19:41:08.138794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:08.138751    5958 scope.go:122] "RemoveContainer" containerID="63c9a08723010ac85b7be05523c764262b2fe1c0eadd10aeeb78d71a1ec50dee"
Oct 31 19:41:08.151007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:08.150982    5958 scope.go:122] "RemoveContainer" containerID="2f9de7c83c96c0e722ee0b869895c19204ac58f99940450b0941b384edb3223e"
Oct 31 19:41:08.224357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:08.223918    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:08.224357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:08.223981    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:09.231570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:09.222985    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:09.231570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:09.223066    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:10.221571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:10.221417    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:10.221571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:10.221480    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:11.227390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:11.225145    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 19:41:11.227390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:11.225223    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 19:41:12.251228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:12.249000    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:12.251228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:12.250923    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:12.257353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:12.257321    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:14.782763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.782731    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"]
Oct 31 19:41:14.808776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.808730    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.817168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.815526    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-fd1c91-openshift-default-dockercfg-hm6l2\""
Oct 31 19:41:14.838194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.838160    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"]
Oct 31 19:41:14.881249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.881203    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.881451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.881259    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.881451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.881368    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.881631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.881612    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.881900 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.881874    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqcs\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.881979 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.881947    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.882102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.882079    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.882142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.882120    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.882187 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.882170    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984126    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984326    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqcs\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984498    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984530    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984595    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984696    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984725    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.984768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984754    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.985016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.984991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.985270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.985246    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.985478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.985320    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.985623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.985350    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.985623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.985584    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.987140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.987116    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:14.987349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:14.987330    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:15.001173 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.001148    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqcs\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:15.011349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.011316    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:15.153772 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.153736    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:15.246459 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.238115    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"]
Oct 31 19:41:15.260005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.259970    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.312863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.312811    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"]
Oct 31 19:41:15.316851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.316758    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317086 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.316914    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317086 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.316976    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317086 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.317074    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.317129    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.317171    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.317210    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.317241    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.317461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.317363    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f59f\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.419902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.419504    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.419902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.419590    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.419902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.419637    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.419902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.419670    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.420386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420172    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.420483 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420379    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f59f\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420529    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420604    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420731    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420794    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420894    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.420897    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.421576    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.422775    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.424593    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.424960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.424792    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.435104 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.435074    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f59f\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.435627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.435602    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.625883 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:41:15.625852    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d701ad4_cb46_46e5_94d4_15a01315912c.slice/crio-cf8fc1f28d526c5b63cb96a34a3d18ff149a3828f088da1a4679a2e9aa9ac50f WatchSource:0}: Error finding container cf8fc1f28d526c5b63cb96a34a3d18ff149a3828f088da1a4679a2e9aa9ac50f: Status 404 returned error can't find the container with id cf8fc1f28d526c5b63cb96a34a3d18ff149a3828f088da1a4679a2e9aa9ac50f
Oct 31 19:41:15.626242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.626220    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"]
Oct 31 19:41:15.628727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.628695    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:15.628810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.628764    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:15.628810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.628799    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:15.632535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.632512    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:15.974168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.974128    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"]
Oct 31 19:41:15.976573 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:41:15.976428    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d0795b_4530_47c7_bacb_c934bdea1f21.slice/crio-4acacb053af1ef39bbcc889a5a35c1fca0a45f4bc3a10b4ec06793c88c55781e WatchSource:0}: Error finding container 4acacb053af1ef39bbcc889a5a35c1fca0a45f4bc3a10b4ec06793c88c55781e: Status 404 returned error can't find the container with id 4acacb053af1ef39bbcc889a5a35c1fca0a45f4bc3a10b4ec06793c88c55781e
Oct 31 19:41:15.981201 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.981161    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:15.981335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.981240    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:15.981335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:15.981271    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:16.638689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:16.638649    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" event={"ID":"34d0795b-4530-47c7-bacb-c934bdea1f21","Type":"ContainerStarted","Data":"46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2"}
Oct 31 19:41:16.638879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:16.638731    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" event={"ID":"34d0795b-4530-47c7-bacb-c934bdea1f21","Type":"ContainerStarted","Data":"4acacb053af1ef39bbcc889a5a35c1fca0a45f4bc3a10b4ec06793c88c55781e"}
Oct 31 19:41:16.656595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:16.655496    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" event={"ID":"9d701ad4-cb46-46e5-94d4-15a01315912c","Type":"ContainerStarted","Data":"8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72"}
Oct 31 19:41:16.656595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:16.655579    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" event={"ID":"9d701ad4-cb46-46e5-94d4-15a01315912c","Type":"ContainerStarted","Data":"cf8fc1f28d526c5b63cb96a34a3d18ff149a3828f088da1a4679a2e9aa9ac50f"}
Oct 31 19:41:16.692568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:16.683835    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podStartSLOduration=1.683813678 podStartE2EDuration="1.683813678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.678018247 +0000 UTC m=+1270.139392863" watchObservedRunningTime="2026-10-31 19:41:16.683813678 +0000 UTC m=+1270.145188294"
Oct 31 19:41:16.722182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:16.722116    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podStartSLOduration=2.722065794 podStartE2EDuration="2.722065794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.721120416 +0000 UTC m=+1270.182495032" watchObservedRunningTime="2026-10-31 19:41:16.722065794 +0000 UTC m=+1270.183440651"
Oct 31 19:41:17.225599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:17.218146    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:17.225599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:17.219138    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:17.225599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:17.219224    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:17.239133 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:17.238661    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:41:17.642657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:17.642166    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:17.663567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:17.642392    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:17.663567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:17.658276    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:18.162027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:18.161647    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:18.162027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:18.161705    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:18.652571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:18.637264    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:18.654042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:18.654001    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:18.996185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:18.996091    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"]
Oct 31 19:41:19.001826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.001798    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.045632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.033849    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e73256-openshift-default-dockercfg-dp7zl\""
Oct 31 19:41:19.083259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.083218    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"]
Oct 31 19:41:19.094751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.090905    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"]
Oct 31 19:41:19.099209 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.099181    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"]
Oct 31 19:41:19.099385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.099313    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.164574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.164082    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:19.164574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.164148    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:19.174569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.174528    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.174751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.174736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.174870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.174859    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.174951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.174941    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.175059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.175050    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.175174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.175165    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.175265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.175257    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.175365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.175355    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl6j5\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.175520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.175497    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.276753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276658    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.276753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276714    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.276753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276736    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276781    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwv4x\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276814    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276861    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276885    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276905    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276934    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.276981    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277031    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277311    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277357    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277371    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl6j5\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277426    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277726    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277745    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.277884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277865    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277914    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.277981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.277960    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.278038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.278021    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.278094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.278057    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.278384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.278365    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.279749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.279729    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.280427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.280405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.291492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.291466    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.292146 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.292125    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl6j5\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.381867    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.382338    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.382370    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.382851    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383178    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383280    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383360    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwv4x\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383461    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383498    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383529    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.383710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.383686    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.384824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.384796    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.384950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.384822    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.385300 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.385250    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.385832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.385756    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.387758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.387739    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.391734 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.391712    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:19.399774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.399751    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwv4x\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.400150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.400133    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.477562 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.476853    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:19.689787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.689745    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:19.690193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.689835    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:19.886317 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.886277    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"]
Oct 31 19:41:19.887281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:19.886922    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" containerID="cri-o://e6e17427989aec2597d54342b905063f79d85608275c1986d78a46e0852d0c03" gracePeriod=30
Oct 31 19:41:20.183296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.183250    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:20.183506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.183317    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:20.462571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.448893    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"]
Oct 31 19:41:20.462571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.460411    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:20.462571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.460481    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:20.462571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.460512    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:20.567724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.567462    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"]
Oct 31 19:41:20.580378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.579911    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:20.580378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.579996    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:20.580378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.580031    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:20.655622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.654534    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:20.655622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.654616    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:20.779340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.777033    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"]
Oct 31 19:41:20.779340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:20.777271    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" podUID="f077dc9b-353f-4590-9cf5-4d406aaca326" containerName="maas-api" containerID="cri-o://5dc3235c2227045ad6d5ef200d20847f6d2fd0f3f73a633deb0d2ef5d9c55e3e" gracePeriod=30
Oct 31 19:41:21.167572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.163018    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.167572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.163094    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:21.243652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.243536    5958 generic.go:356] "Generic (PLEG): container finished" podID="f077dc9b-353f-4590-9cf5-4d406aaca326" containerID="5dc3235c2227045ad6d5ef200d20847f6d2fd0f3f73a633deb0d2ef5d9c55e3e" exitCode=0
Oct 31 19:41:21.244302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.244274    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" event={"ID":"f077dc9b-353f-4590-9cf5-4d406aaca326","Type":"ContainerDied","Data":"5dc3235c2227045ad6d5ef200d20847f6d2fd0f3f73a633deb0d2ef5d9c55e3e"}
Oct 31 19:41:21.257579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.248316    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" event={"ID":"3e8f67c6-c9e6-409b-9808-192635401cc5","Type":"ContainerStarted","Data":"3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e"}
Oct 31 19:41:21.257579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.248374    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" event={"ID":"3e8f67c6-c9e6-409b-9808-192635401cc5","Type":"ContainerStarted","Data":"0535bfe4ad158976786f4ffa553a9ddd7ba1f999d2f50c5d1ef4e9736ee2674d"}
Oct 31 19:41:21.362317 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.362264    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" event={"ID":"3b16dfa4-0e73-4df0-ac2d-a94961f993a8","Type":"ContainerStarted","Data":"b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df"}
Oct 31 19:41:21.362617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.362591    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" event={"ID":"3b16dfa4-0e73-4df0-ac2d-a94961f993a8","Type":"ContainerStarted","Data":"54938f22fe9d6d65165e56d38116184a75979e45183f9a87a70f3e139fcab745"}
Oct 31 19:41:21.396880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.396849    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:21.529710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.460000    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.529710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.460072    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:21.529710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.475508    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podStartSLOduration=3.47547525 podStartE2EDuration="3.47547525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:21.419005553 +0000 UTC m=+1274.880380182" watchObservedRunningTime="2026-10-31 19:41:21.47547525 +0000 UTC m=+1274.936849867"
Oct 31 19:41:21.529710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.486726    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:21.529710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.503138    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.529710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.503205    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:21.639055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.637032    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.639055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.637100    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:21.661866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.656782    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podStartSLOduration=3.656758903 podStartE2EDuration="3.656758903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:21.650760144 +0000 UTC m=+1275.112134756" watchObservedRunningTime="2026-10-31 19:41:21.656758903 +0000 UTC m=+1275.118133518"
Oct 31 19:41:21.855944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.797910    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:41:21.916600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:21.912085    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:22.191785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.191716    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:22.192009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.191782    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:22.224653 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.224058    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:41:22.419523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.391722    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:22.466005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.465194    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:22.466005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.465258    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:22.487896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.487776    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:22.487896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.487835    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:22.653375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.652900    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:22.653375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:22.652969    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:23.188565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.185589    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.188565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.185653    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:23.273143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.271502    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:23.591684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.565963    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.591684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.566029    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:23.591684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.566692    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:23.750227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.750173    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.750527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.750479    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:23.766879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.766845    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:41:23.790626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.775176    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:41:23.861187 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.861086    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:23.861888 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:23.778283    5958 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.115s"
Oct 31 19:41:23.865570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.862474    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:41:23.865743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.865723    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"]
Oct 31 19:41:23.906572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.866054    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" containerID="cri-o://ab59594e92d15e545d00edc80d8ad96def129745b599d645c7889cdd8f5d05da" gracePeriod=30
Oct 31 19:41:23.979402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.974464    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"]
Oct 31 19:41:23.979402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:23.974761    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" containerID="cri-o://f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa" gracePeriod=30
Oct 31 19:41:24.060807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.045119    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:41:24.097701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.076290    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:41:24.098057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.096801    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:24.118775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.114395    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"
Oct 31 19:41:24.130672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.130622    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" event={"ID":"f077dc9b-353f-4590-9cf5-4d406aaca326","Type":"ContainerDied","Data":"576339f2ea9892ad3580f85199f61afc9d2ae692301c5e01b3139b0a9c2ee990"}
Oct 31 19:41:24.130959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.130940    5958 scope.go:122] "RemoveContainer" containerID="5dc3235c2227045ad6d5ef200d20847f6d2fd0f3f73a633deb0d2ef5d9c55e3e"
Oct 31 19:41:24.320575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.297662    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"]
Oct 31 19:41:24.403260 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.403162    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" containerID="cri-o://7b41f43e1c9d3a09eb11442f0d1404ae721163385d8afa15872d0b067170b161" gracePeriod=30
Oct 31 19:41:24.418861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.322496    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:41:24.454870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.419239    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 31 19:41:24.454870 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.444003    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 31 19:41:24.492399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.423648    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:24.492399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.472209    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:24.492399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.472257    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:24.531576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.528680    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:41:24.569971 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.569914    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:24.575621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.575585    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:24.785973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.613176    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:24.786207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.648652    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca\") pod \"f077dc9b-353f-4590-9cf5-4d406aaca326\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") "
Oct 31 19:41:24.786469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.786449    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls\") pod \"f077dc9b-353f-4590-9cf5-4d406aaca326\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") "
Oct 31 19:41:24.786724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.786703    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs\") pod \"f077dc9b-353f-4590-9cf5-4d406aaca326\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") "
Oct 31 19:41:24.802601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.787130    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7\" (UniqueName: \"kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7\") pod \"f077dc9b-353f-4590-9cf5-4d406aaca326\" (UID: \"f077dc9b-353f-4590-9cf5-4d406aaca326\") "
Oct 31 19:41:24.805301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.676329    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:24.824596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.677344    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca" pod "f077dc9b-353f-4590-9cf5-4d406aaca326" (UID: "f077dc9b-353f-4590-9cf5-4d406aaca326"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:24.824799 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.704703    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 31 19:41:24.824997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.824966    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 31 19:41:24.841050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.836657    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls" pod "f077dc9b-353f-4590-9cf5-4d406aaca326" (UID: "f077dc9b-353f-4590-9cf5-4d406aaca326"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:24.864078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.862226    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:41:24.948206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:24.781536    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:25.002725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.002687    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:25.002982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.002949    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:25.250295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.250248    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:25.260872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.260824    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l" podUID="f077dc9b-353f-4590-9cf5-4d406aaca326" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": context deadline exceeded"
Oct 31 19:41:25.434612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.299608    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs" pod "f077dc9b-353f-4590-9cf5-4d406aaca326" (UID: "f077dc9b-353f-4590-9cf5-4d406aaca326"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:41:25.435158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.326310    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f077dc9b-353f-4590-9cf5-4d406aaca326-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.435301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.435284    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.460566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.460130    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:25.491029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.484530    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:25.491029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.484615    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:25.491029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.485958    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:25.491029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.486001    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.506071    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.506148    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.506306    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.506351    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.530003    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.531609    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.539394    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f077dc9b-353f-4590-9cf5-4d406aaca326-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.546589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.545654    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:25.563975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.563930    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7" pod "f077dc9b-353f-4590-9cf5-4d406aaca326" (UID: "f077dc9b-353f-4590-9cf5-4d406aaca326"). InnerVolumeSpecName "kube-api-access-2z6b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:25.572811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.572765    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:25.598327 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.588830    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:41:25.598327 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.590162    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 19:41:25.663689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.660351    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:25.695263 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.695181    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:41:25.743450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.743309    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6b7\" (UniqueName: \"kubernetes.io/projected/f077dc9b-353f-4590-9cf5-4d406aaca326-kube-api-access-2z6b7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.898006 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.897966    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:41:25.981175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.981080    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:26.002226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:25.996823    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:26.060604 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.041077    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:26.205468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.204654    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:41:26.251962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.251878    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"]
Oct 31 19:41:26.544779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.544685    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:26.545360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.545320    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:26.571568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.546386    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:26.572400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.572350    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:26.794843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:26.794732    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:27.020055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.019990    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:27.031835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.024496    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:41:27.032580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.032497    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 31 19:41:27.181454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.181413    5958 generic.go:356] "Generic (PLEG): container finished" podID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerID="e6e17427989aec2597d54342b905063f79d85608275c1986d78a46e0852d0c03" exitCode=0
Oct 31 19:41:27.192997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.192949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:41:27.192997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.192994    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" event={"ID":"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a","Type":"ContainerDied","Data":"e6e17427989aec2597d54342b905063f79d85608275c1986d78a46e0852d0c03"}
Oct 31 19:41:27.326576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.241791    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:41:27.453628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.453185    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b21c-6999489686-5pd7l"]
Oct 31 19:41:27.666413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.666354    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:41:27.833621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.821689    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:27.853323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.853275    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:27.854122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.844111    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:27.910324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.908492    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:41:27.910324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.908575    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:27.911087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.911058    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 19:41:27.911284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:27.911262    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:28.268569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.242584    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:28.339081 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.249153    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Oct 31 19:41:28.340206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.340176    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:41:28.803516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.315926    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:28.818377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.811879    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:28.843986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.801032    5958 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:41:28.844278 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.844246    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:28.856396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.803471    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:28.866739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.857049    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:28.866739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.860761    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:28.866739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.860898    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:28.866739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.860927    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:28.947788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.932575    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:41:28.983658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.983600    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:28.991574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.990847    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:28.994474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.992621    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:28.994474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.992748    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:28.994474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:28.993001    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.002195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.002168    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:29.063025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.062926    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:29.081713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.081680    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"
Oct 31 19:41:29.096012 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.095413    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.096366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.096327    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.096674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.096627    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.096924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.096888    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.097118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.097083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.097377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.097346    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert\") pod \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\" (UID: \"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a\") "
Oct 31 19:41:29.098701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.098682    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.100255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.100233    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:29.125010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.124792    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:29.125434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.125248    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:29.127217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.125642    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:29.189646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.188109    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:29.198733 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.192731    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "kube-api-access-vd6vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:29.378585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.378525    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381375    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381412    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381432    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6vx\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-kube-api-access-vd6vx\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381450    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381464    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381479    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.383597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.381493    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.425586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.386809    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:41:29.425586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.386867    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:29.425586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.413441    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:29.425586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.413496    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:29.479718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.479676    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:29.479991 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.479955    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:29.481729 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.481699    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f077dc9b-353f-4590-9cf5-4d406aaca326" path="/var/lib/kubelet/pods/f077dc9b-353f-4590-9cf5-4d406aaca326/volumes"
Oct 31 19:41:29.533152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.533105    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52" event={"ID":"4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a","Type":"ContainerDied","Data":"a9b63f53783f99b85b71926a60fcb5a3371746b9c39ebe3a4a5e96b49357d152"}
Oct 31 19:41:29.534840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.534813    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:41:29.535043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.535031    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:41:29.535193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.535180    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:41:29.535302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.535293    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:41:29.535537 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.531319    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token" pod "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" (UID: "4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:29.535681 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.535670    5958 scope.go:122] "RemoveContainer" containerID="e6e17427989aec2597d54342b905063f79d85608275c1986d78a46e0852d0c03"
Oct 31 19:41:29.555023 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:29.554972    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d3474d_ff6c_488e_9c18_0ded72b45f30.slice/crio-f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 19:41:29.562665 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:29.562276    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d3474d_ff6c_488e_9c18_0ded72b45f30.slice/crio-f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 19:41:29.660622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.614535    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.993361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:29.993054    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:30.028658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.023850    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:41:30.077459 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.076774    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:41:30.086674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.078236    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:30.086674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.079299    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:41:30.197607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.197566    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:41:30.379571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.379524    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:30.391587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.391103    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.392113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.392066    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.392352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.392333    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.392519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.392491    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.393023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.392988    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.393259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.393241    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.393751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.393716    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.394040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.394002    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.394274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.394256    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k\") pod \"71d3474d-ff6c-488e-9c18-0ded72b45f30\" (UID: \"71d3474d-ff6c-488e-9c18-0ded72b45f30\") "
Oct 31 19:41:30.399290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.399264    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:30.400009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.399974    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:30.400379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.400359    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:30.400593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.400559    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:30.401001 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.400971    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:30.552568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.528384    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 19:41:30.552876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.552833    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 19:41:30.553699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.553646    5958 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 19:41:30.553968 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.553935    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 19:41:30.687061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.686970    5958 generic.go:356] "Generic (PLEG): container finished" podID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerID="7b41f43e1c9d3a09eb11442f0d1404ae721163385d8afa15872d0b067170b161" exitCode=0
Oct 31 19:41:30.758574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.758515    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.758789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.758750    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.758874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.758789    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d3474d-ff6c-488e-9c18-0ded72b45f30-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.758874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.758806    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.758874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.758822    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.769138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.769107    5958 generic.go:356] "Generic (PLEG): container finished" podID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerID="f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa" exitCode=0
Oct 31 19:41:30.769660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.769640    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"
Oct 31 19:41:30.786599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.786529    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" event={"ID":"933a8af3-fa2c-4bd9-a51c-9d2295f28c19","Type":"ContainerDied","Data":"7b41f43e1c9d3a09eb11442f0d1404ae721163385d8afa15872d0b067170b161"}
Oct 31 19:41:30.786906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.786881    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd" event={"ID":"71d3474d-ff6c-488e-9c18-0ded72b45f30","Type":"ContainerDied","Data":"f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa"}
Oct 31 19:41:30.787106 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.787070    5958 scope.go:122] "RemoveContainer" containerID="f4350b2b8ae4ac827fe6a317e0bf23a62bcaf89948f01d7358ce1f3cb6bcfeaa"
Oct 31 19:41:30.790432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.788942    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:30.790432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.789059    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "kube-api-access-9p58k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:30.802515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.800728    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:30.850842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.850788    5958 generic.go:356] "Generic (PLEG): container finished" podID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerID="ab59594e92d15e545d00edc80d8ad96def129745b599d645c7889cdd8f5d05da" exitCode=0
Oct 31 19:41:30.851973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.851926    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" event={"ID":"ae6edde9-79e2-4fb9-ac86-d0c7626c615a","Type":"ContainerDied","Data":"ab59594e92d15e545d00edc80d8ad96def129745b599d645c7889cdd8f5d05da"}
Oct 31 19:41:30.859118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.859082    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token" pod "71d3474d-ff6c-488e-9c18-0ded72b45f30" (UID: "71d3474d-ff6c-488e-9c18-0ded72b45f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:30.892323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.892283    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p58k\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-kube-api-access-9p58k\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.892562 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.892525    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.892715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.892699    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.892812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.892798    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d3474d-ff6c-488e-9c18-0ded72b45f30-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:30.920900 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.920863    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"]
Oct 31 19:41:30.922838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922526    5958 cpu_manager.go:414] "containerMap: removing container" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy"
Oct 31 19:41:30.922838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922596    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy"
Oct 31 19:41:30.922838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922700    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f077dc9b-353f-4590-9cf5-4d406aaca326" containerName="maas-api"
Oct 31 19:41:30.922838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922709    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f077dc9b-353f-4590-9cf5-4d406aaca326" containerName="maas-api"
Oct 31 19:41:30.922838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922727    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy"
Oct 31 19:41:30.922838 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922735    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy"
Oct 31 19:41:30.923272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.922994    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f077dc9b-353f-4590-9cf5-4d406aaca326" containerName="maas-api"
Oct 31 19:41:30.923272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.923008    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" containerName="istio-proxy"
Oct 31 19:41:30.923272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.923022    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" containerName="istio-proxy"
Oct 31 19:41:30.932581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.929320    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:30.993779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:30.993561    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"]
Oct 31 19:41:31.000533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.000160    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.000533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.000489    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.000784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.000755    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.000924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.000905    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.001137 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.001115    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.032754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.032727    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"]
Oct 31 19:41:31.043214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.043187    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-cnj52"]
Oct 31 19:41:31.056250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.056221    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:41:31.112754 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.112722    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.113066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.113041    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gbm7\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.113324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.113304    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.113517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.113500    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.113736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.113720    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.113973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.113955    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.114227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.114194    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.114697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.114675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.114845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.114828    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.115522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.115501    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.115968 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.115944    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.117034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.117013    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.119679 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.119657    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.147641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.147601    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.150460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.150429    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:41:31.177182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.175170    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"]
Oct 31 19:41:31.190764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.190677    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-2bcjd"]
Oct 31 19:41:31.616740 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.616673    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:31.618033 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.617996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.618645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.618469    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:31.621749 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.619288    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.622043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.622023    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.622189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.622172    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.622322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.622307    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.622471 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.622455    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.622607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.622593    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.628592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.628527    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.628875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.628837    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.629129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.629112    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.629332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.629297    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.629674 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.629657    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.629825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.629808    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.630108 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.630072    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.630345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.630329    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.630595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.630581    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.630872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.630837    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token\") pod \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\" (UID: \"ae6edde9-79e2-4fb9-ac86-d0c7626c615a\") "
Oct 31 19:41:31.638247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.638184    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket\") pod \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\" (UID: \"933a8af3-fa2c-4bd9-a51c-9d2295f28c19\") "
Oct 31 19:41:31.640595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.640529    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.664706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.664656    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.664946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.664925    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbm7\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.665175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.665153    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.665476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.623563    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.665603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.623820    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.665705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.642657    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.665777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.648240    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.665859 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.648903    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.665938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.649145    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:31.666045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.649186    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.666121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.649528    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:31.666190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.650142    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.666271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.653954    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:31.666351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.659097    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.666414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.642333    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.666512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.623745    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:41:31.667557 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.667503    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.669415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.667650    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:31.671220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.671042    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.671759 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.671732    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:31.672050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.672024    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "kube-api-access-x9kk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:31.675107 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.675072    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token" pod "ae6edde9-79e2-4fb9-ac86-d0c7626c615a" (UID: "ae6edde9-79e2-4fb9-ac86-d0c7626c615a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:31.676328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.676296    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:31.682834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.682771    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9" pod "933a8af3-fa2c-4bd9-a51c-9d2295f28c19" (UID: "933a8af3-fa2c-4bd9-a51c-9d2295f28c19"). InnerVolumeSpecName "kube-api-access-ld5q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:31.698257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.698180    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.700686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.700641    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbm7\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767264    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9kk2\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-kube-api-access-x9kk2\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767310    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767326    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767340    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767354    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767370    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767384    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767398    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767411    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767425    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767438    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5q9\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-kube-api-access-ld5q9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767453    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767466    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767480    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767493    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767507    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/933a8af3-fa2c-4bd9-a51c-9d2295f28c19-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767520    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.767589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.767534    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6edde9-79e2-4fb9-ac86-d0c7626c615a-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:31.861035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.860991    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:31.862111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.862019    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:31.863957 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.863878    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:41:31.872607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.870588    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"
Oct 31 19:41:31.872607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.871476    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89" event={"ID":"933a8af3-fa2c-4bd9-a51c-9d2295f28c19","Type":"ContainerDied","Data":"e54cb31c5ea6179590a1d443e557a50bba8458c9ad347d0a59f31f43468b38ff"}
Oct 31 19:41:31.872607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.871567    5958 scope.go:122] "RemoveContainer" containerID="7b41f43e1c9d3a09eb11442f0d1404ae721163385d8afa15872d0b067170b161"
Oct 31 19:41:31.883810 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.881743    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"
Oct 31 19:41:31.884812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.882539    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg" event={"ID":"ae6edde9-79e2-4fb9-ac86-d0c7626c615a","Type":"ContainerDied","Data":"9501ecffef07712c9adf4ddbf9bd38dc396af19da7f3d6f293676901b72468c1"}
Oct 31 19:41:31.915716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.915689    5958 scope.go:122] "RemoveContainer" containerID="ab59594e92d15e545d00edc80d8ad96def129745b599d645c7889cdd8f5d05da"
Oct 31 19:41:31.980567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:31.979907    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:32.078734 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.075201    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"]
Oct 31 19:41:32.129016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.124735    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tpv89"]
Oct 31 19:41:32.141689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.141660    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"]
Oct 31 19:41:32.151079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.151035    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w2tzg"]
Oct 31 19:41:32.425654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.425626    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"]
Oct 31 19:41:32.426510 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:41:32.426483    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3935dfcf_fd68_4674_be17_5b016abd4500.slice/crio-c52c7a34cc9e7bcc1a52b3f475551ccfe1b0e2f68aafe4146e8c6e5a867bfdc4 WatchSource:0}: Error finding container c52c7a34cc9e7bcc1a52b3f475551ccfe1b0e2f68aafe4146e8c6e5a867bfdc4: Status 404 returned error can't find the container with id c52c7a34cc9e7bcc1a52b3f475551ccfe1b0e2f68aafe4146e8c6e5a867bfdc4
Oct 31 19:41:32.429491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.429456    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:32.429652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.429554    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:32.429652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.429605    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:32.573190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.573144    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"]
Oct 31 19:41:32.573474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.573439    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy" containerID="cri-o://b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8" gracePeriod=30
Oct 31 19:41:32.586648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.586279    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"]
Oct 31 19:41:32.605509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.605472    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"]
Oct 31 19:41:32.605996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.605960    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy" containerID="cri-o://46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2" gracePeriod=30
Oct 31 19:41:32.889692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.889478    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a" path="/var/lib/kubelet/pods/4ccd8d55-ebae-440c-8a1a-9b1e51d61b6a/volumes"
Oct 31 19:41:32.890634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.890609    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d3474d-ff6c-488e-9c18-0ded72b45f30" path="/var/lib/kubelet/pods/71d3474d-ff6c-488e-9c18-0ded72b45f30/volumes"
Oct 31 19:41:32.892081 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.892058    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" path="/var/lib/kubelet/pods/933a8af3-fa2c-4bd9-a51c-9d2295f28c19/volumes"
Oct 31 19:41:32.977600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:32.936571    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" path="/var/lib/kubelet/pods/ae6edde9-79e2-4fb9-ac86-d0c7626c615a/volumes"
Oct 31 19:41:33.090669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.090625    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" event={"ID":"3935dfcf-fd68-4674-be17-5b016abd4500","Type":"ContainerStarted","Data":"e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968"}
Oct 31 19:41:33.091188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.090717    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" event={"ID":"3935dfcf-fd68-4674-be17-5b016abd4500","Type":"ContainerStarted","Data":"c52c7a34cc9e7bcc1a52b3f475551ccfe1b0e2f68aafe4146e8c6e5a867bfdc4"}
Oct 31 19:41:33.091188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.091110    5958 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" secret="" err="secret \"e2e-api-b-d4b21c-openshift-default-dockercfg-k8zbt\" not found"
Oct 31 19:41:33.122905 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.122686    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" podStartSLOduration=3.122668014 podStartE2EDuration="3.122668014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:33.121019807 +0000 UTC m=+1286.582394426" watchObservedRunningTime="2026-10-31 19:41:33.122668014 +0000 UTC m=+1286.584042629"
Oct 31 19:41:33.980936 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.980895    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:33.982622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.982579    5958 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 19:41:33.982785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:33.982647    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" podUID="3935dfcf-fd68-4674-be17-5b016abd4500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 19:41:34.290867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:34.290768    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" podUID="3935dfcf-fd68-4674-be17-5b016abd4500" containerName="istio-proxy" containerID="cri-o://e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968" gracePeriod=30
Oct 31 19:41:37.972489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:37.972458    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:38.032386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.029408    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:41:38.071097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071057    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071295    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071331    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071471    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071501    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071586    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071616    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071728    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.071914 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071815    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.071914 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.071894    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token\") pod \"34d0795b-4530-47c7-bacb-c934bdea1f21\" (UID: \"34d0795b-4530-47c7-bacb-c934bdea1f21\") "
Oct 31 19:41:38.073294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.073244    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.074099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.073908    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.074099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.073927    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:38.074522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.074468    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.074755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.074733    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.075354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.075203    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "kube-api-access-2f59f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.085038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.085008    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:38.085164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.085040    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.087025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.087001    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy" pod "34d0795b-4530-47c7-bacb-c934bdea1f21" (UID: "34d0795b-4530-47c7-bacb-c934bdea1f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.174817 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.174776    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.174986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.174878    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.174986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.174929    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.174996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175046    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175072    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175092    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175124    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.175172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175140    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175243    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data\") pod \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\" (UID: \"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262\") "
Oct 31 19:41:38.175372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175292    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.175512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175488    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.175605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175517    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.175754    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176420    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176445    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176461    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176475    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176489    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d0795b-4530-47c7-bacb-c934bdea1f21-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176503    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f59f\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-kube-api-access-2f59f\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176520    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176533    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176567    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176581    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176596    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d0795b-4530-47c7-bacb-c934bdea1f21-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176611    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d0795b-4530-47c7-bacb-c934bdea1f21-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.176670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.176625    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.177475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.177289    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.177716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.177691    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:38.177813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.177734    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.177858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.177843    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz" pod "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" (UID: "3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262"). InnerVolumeSpecName "kube-api-access-fm8hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.278920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.278877    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.278920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.278918    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8hz\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-kube-api-access-fm8hz\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.278920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.278928    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.279167 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.278937    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.328036 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.328000    5958 generic.go:356] "Generic (PLEG): container finished" podID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerID="b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8" exitCode=0
Oct 31 19:41:38.328219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.328053    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"
Oct 31 19:41:38.328219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.328089    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" event={"ID":"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262","Type":"ContainerDied","Data":"b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8"}
Oct 31 19:41:38.328219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.328163    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z" event={"ID":"3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262","Type":"ContainerDied","Data":"e39bfa83b7996bf54bcbd3792769bcaf3af021a4f6275a46740ef5dc0893e53e"}
Oct 31 19:41:38.328219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.328192    5958 scope.go:122] "RemoveContainer" containerID="b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8"
Oct 31 19:41:38.329919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.329821    5958 generic.go:356] "Generic (PLEG): container finished" podID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerID="46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2" exitCode=0
Oct 31 19:41:38.329919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.329876    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"
Oct 31 19:41:38.329919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.329910    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" event={"ID":"34d0795b-4530-47c7-bacb-c934bdea1f21","Type":"ContainerDied","Data":"46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2"}
Oct 31 19:41:38.330167 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.329963    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs" event={"ID":"34d0795b-4530-47c7-bacb-c934bdea1f21","Type":"ContainerDied","Data":"4acacb053af1ef39bbcc889a5a35c1fca0a45f4bc3a10b4ec06793c88c55781e"}
Oct 31 19:41:38.346704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.346668    5958 scope.go:122] "RemoveContainer" containerID="b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8"
Oct 31 19:41:38.347048 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:38.347027    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8\": container with ID starting with b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8 not found: ID does not exist" containerID="b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8"
Oct 31 19:41:38.347145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.347063    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8"} err="failed to get container status \"b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8\": rpc error: code = NotFound desc = could not find container \"b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8\": container with ID starting with b3766526bf9530f7f609ab4b03578ca421ce81448140afbbae1b67b8d965b6c8 not found: ID does not exist"
Oct 31 19:41:38.347145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.347088    5958 scope.go:122] "RemoveContainer" containerID="46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2"
Oct 31 19:41:38.360044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.359931    5958 scope.go:122] "RemoveContainer" containerID="46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2"
Oct 31 19:41:38.360329 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:38.360299    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2\": container with ID starting with 46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2 not found: ID does not exist" containerID="46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2"
Oct 31 19:41:38.360440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.360342    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2"} err="failed to get container status \"46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2\": rpc error: code = NotFound desc = could not find container \"46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2\": container with ID starting with 46bd411aea135b25685427699fa61f48537490cc8982d10683213163334ac5c2 not found: ID does not exist"
Oct 31 19:41:38.361975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.361956    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"]
Oct 31 19:41:38.367338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.367317    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-wqr6z"]
Oct 31 19:41:38.382234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.382182    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"]
Oct 31 19:41:38.398873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.398831    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-xcrcs"]
Oct 31 19:41:38.672561 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.672512    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" path="/var/lib/kubelet/pods/34d0795b-4530-47c7-bacb-c934bdea1f21/volumes"
Oct 31 19:41:38.673397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:38.673375    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" path="/var/lib/kubelet/pods/3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262/volumes"
Oct 31 19:41:40.081414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.081383    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:40.168664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168522    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.168664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168624    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.168664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168658    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.168977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168693    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.168977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168777    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.168977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168834    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:40.168977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.168920    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.169192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:40.169192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169039    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.169192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169127    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.169363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169287    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy\") pod \"3935dfcf-fd68-4674-be17-5b016abd4500\" (UID: \"3935dfcf-fd68-4674-be17-5b016abd4500\") "
Oct 31 19:41:40.169363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169295    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:40.169535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169512    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:40.169751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.169698    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:40.170300 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.170278    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.170438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.170421    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3935dfcf-fd68-4674-be17-5b016abd4500-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.170538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.170525    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.170704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.170688    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.170817 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.170802    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.172765 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.172723    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:40.172880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.172756    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "kube-api-access-8gbm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:40.172880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.172824    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:40.172880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.172819    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy" pod "3935dfcf-fd68-4674-be17-5b016abd4500" (UID: "3935dfcf-fd68-4674-be17-5b016abd4500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:40.272562 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.272510    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.272769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.272578    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3935dfcf-fd68-4674-be17-5b016abd4500-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.272769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.272591    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3935dfcf-fd68-4674-be17-5b016abd4500-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.272769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.272600    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gbm7\" (UniqueName: \"kubernetes.io/projected/3935dfcf-fd68-4674-be17-5b016abd4500-kube-api-access-8gbm7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:40.353523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.353490    5958 generic.go:356] "Generic (PLEG): container finished" podID="3935dfcf-fd68-4674-be17-5b016abd4500" containerID="e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968" exitCode=0
Oct 31 19:41:40.353713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.353535    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"
Oct 31 19:41:40.353713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.353570    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" event={"ID":"3935dfcf-fd68-4674-be17-5b016abd4500","Type":"ContainerDied","Data":"e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968"}
Oct 31 19:41:40.353793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.353705    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn" event={"ID":"3935dfcf-fd68-4674-be17-5b016abd4500","Type":"ContainerDied","Data":"c52c7a34cc9e7bcc1a52b3f475551ccfe1b0e2f68aafe4146e8c6e5a867bfdc4"}
Oct 31 19:41:40.353793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.353740    5958 scope.go:122] "RemoveContainer" containerID="e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968"
Oct 31 19:41:40.366952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.366929    5958 scope.go:122] "RemoveContainer" containerID="e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968"
Oct 31 19:41:40.367271 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:40.367246    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968\": container with ID starting with e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968 not found: ID does not exist" containerID="e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968"
Oct 31 19:41:40.367374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.367278    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968"} err="failed to get container status \"e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968\": rpc error: code = NotFound desc = could not find container \"e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968\": container with ID starting with e9e3b14ce4c393fab5f858fff8a71cde40d426510a599af3839c00da82702968 not found: ID does not exist"
Oct 31 19:41:40.383034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.383004    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"]
Oct 31 19:41:40.387425 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.387394    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-cg6tn"]
Oct 31 19:41:40.670216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:40.670174    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3935dfcf-fd68-4674-be17-5b016abd4500" path="/var/lib/kubelet/pods/3935dfcf-fd68-4674-be17-5b016abd4500/volumes"
Oct 31 19:41:44.800144 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.800093    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"]
Oct 31 19:41:44.801079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801062    5958 cpu_manager.go:414] "containerMap: removing container" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy"
Oct 31 19:41:44.801079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801080    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801093    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3935dfcf-fd68-4674-be17-5b016abd4500" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801098    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3935dfcf-fd68-4674-be17-5b016abd4500" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801107    5958 cpu_manager.go:414] "containerMap: removing container" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801112    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801121    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801127    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801171    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy"
Oct 31 19:41:44.801188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801176    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy"
Oct 31 19:41:44.801427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801307    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="933a8af3-fa2c-4bd9-a51c-9d2295f28c19" containerName="istio-proxy"
Oct 31 19:41:44.801427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801317    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3935dfcf-fd68-4674-be17-5b016abd4500" containerName="istio-proxy"
Oct 31 19:41:44.801427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801323    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d0795b-4530-47c7-bacb-c934bdea1f21" containerName="istio-proxy"
Oct 31 19:41:44.801427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801331    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca7e393-c3b3-4f6a-9ec8-8cfeb249e262" containerName="istio-proxy"
Oct 31 19:41:44.801427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.801343    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6edde9-79e2-4fb9-ac86-d0c7626c615a" containerName="istio-proxy"
Oct 31 19:41:44.806775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.806751    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.853496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.853470    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"]
Oct 31 19:41:44.958579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.951803    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.958919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.958881    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d458n\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.959193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.959175    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.959318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.959303    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.968250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.959482    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.972714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.972675    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.978629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.976668    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.978924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.978886    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:44.979198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:44.979167    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.080930    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.080994    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081076    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081156    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081202    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081252    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081356    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081450    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081607    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.081692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.081637    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d458n\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.082071 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.082043    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.082220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.082143    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.082220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.082194    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.082370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.082348    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.084111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.084089    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.084625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.084603    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.095045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.095022    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.095269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.095247    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d458n\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.152472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.152437    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"]
Oct 31 19:41:45.249600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.249533    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:45.863111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.859865    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"]
Oct 31 19:41:45.869628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.869513    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:45.869964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.869918    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:45.870085 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:45.869976    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:41:46.588035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:46.587995    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" event={"ID":"9784b382-3851-48b5-bf8f-f176fea1d4b7","Type":"ContainerStarted","Data":"c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b"}
Oct 31 19:41:46.588240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:46.588057    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" event={"ID":"9784b382-3851-48b5-bf8f-f176fea1d4b7","Type":"ContainerStarted","Data":"e6801a06fd903ae193980f17552090e1cf5ab00f99f9e29040a8970ea0b18999"}
Oct 31 19:41:46.620480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:46.620428    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podStartSLOduration=2.620413322 podStartE2EDuration="2.620413322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:46.614752989 +0000 UTC m=+1300.076127604" watchObservedRunningTime="2026-10-31 19:41:46.620413322 +0000 UTC m=+1300.081787936"
Oct 31 19:41:47.253742 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:47.253709    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:47.254425 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:47.254401    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:47.254721 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:47.254691    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:47.369501 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:41:47.367807    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:41:48.250124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:48.250084    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:48.250323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:48.250141    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:49.250332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:49.250300    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:49.250760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:49.250354    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:50.152950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.152902    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9" podUID="ef822f74-c7b1-4f26-bc53-4b73a9991643" containerName="payload-pre-processing" containerID="cri-o://443eec9a7947681501c564f5e6d32783c8e560565f6e8f8ce6ac5422dc5e1ab2" gracePeriod=25
Oct 31 19:41:50.325711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.325670    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:50.326319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.326269    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:50.685163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.685129    5958 generic.go:356] "Generic (PLEG): container finished" podID="ef822f74-c7b1-4f26-bc53-4b73a9991643" containerID="443eec9a7947681501c564f5e6d32783c8e560565f6e8f8ce6ac5422dc5e1ab2" exitCode=0
Oct 31 19:41:50.685361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.685240    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9" event={"ID":"ef822f74-c7b1-4f26-bc53-4b73a9991643","Type":"ContainerDied","Data":"443eec9a7947681501c564f5e6d32783c8e560565f6e8f8ce6ac5422dc5e1ab2"}
Oct 31 19:41:50.744391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.744312    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:41:50.752191 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.752165    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume\") pod \"ef822f74-c7b1-4f26-bc53-4b73a9991643\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") "
Oct 31 19:41:50.752341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.752332    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx\") pod \"ef822f74-c7b1-4f26-bc53-4b73a9991643\" (UID: \"ef822f74-c7b1-4f26-bc53-4b73a9991643\") "
Oct 31 19:41:50.752516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.752494    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume" pod "ef822f74-c7b1-4f26-bc53-4b73a9991643" (UID: "ef822f74-c7b1-4f26-bc53-4b73a9991643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:50.753065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.753044    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef822f74-c7b1-4f26-bc53-4b73a9991643-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:50.754855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.754833    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx" pod "ef822f74-c7b1-4f26-bc53-4b73a9991643" (UID: "ef822f74-c7b1-4f26-bc53-4b73a9991643"). InnerVolumeSpecName "kube-api-access-rz2rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:50.854960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:50.854918    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2rx\" (UniqueName: \"kubernetes.io/projected/ef822f74-c7b1-4f26-bc53-4b73a9991643-kube-api-access-rz2rx\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:51.250283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.250249    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:51.250444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.250304    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:51.695372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.695338    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"
Oct 31 19:41:51.695855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.695377    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9" event={"ID":"ef822f74-c7b1-4f26-bc53-4b73a9991643","Type":"ContainerDied","Data":"7c0d3a0648b10d33aa7076c18ef66251bf88210b0beda5557099f12696a8261c"}
Oct 31 19:41:51.695855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.695436    5958 scope.go:122] "RemoveContainer" containerID="443eec9a7947681501c564f5e6d32783c8e560565f6e8f8ce6ac5422dc5e1ab2"
Oct 31 19:41:51.888686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.888639    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"]
Oct 31 19:41:51.948727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:51.936560    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e73256-b9d6b9887-62lj9"]
Oct 31 19:41:52.250761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:52.250678    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:52.251044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:52.251014    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:52.867760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:52.864659    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef822f74-c7b1-4f26-bc53-4b73a9991643" path="/var/lib/kubelet/pods/ef822f74-c7b1-4f26-bc53-4b73a9991643/volumes"
Oct 31 19:41:53.263393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:53.263010    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 31 19:41:53.263806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:53.263698    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 31 19:41:54.255575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:54.255530    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:54.256053 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:54.256036    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:54.256857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:54.256842    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:41:55.804216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:41:55.804179    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:42:01.679599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:01.664114    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:01.679599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:01.664476    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:01.979802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:01.977966    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:17.337218 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:42:17.337171    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:42:21.288638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.288598    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"]
Oct 31 19:42:21.292743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.292710    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ef822f74-c7b1-4f26-bc53-4b73a9991643" containerName="payload-pre-processing"
Oct 31 19:42:21.292955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.292928    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef822f74-c7b1-4f26-bc53-4b73a9991643" containerName="payload-pre-processing"
Oct 31 19:42:21.293229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.293210    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef822f74-c7b1-4f26-bc53-4b73a9991643" containerName="payload-pre-processing"
Oct 31 19:42:21.304572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.302351    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.313958 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.313929    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a813e2-serving-cert\""
Oct 31 19:42:21.338708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.338631    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"]
Oct 31 19:42:21.461119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.461081    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.461327 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.461127    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjgc\" (UniqueName: \"kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.461401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.461345    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.461401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.461384    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.563206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.563117    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.563206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.563152    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjgc\" (UniqueName: \"kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.563402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.563290    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.563402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.563314    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.563580 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:42:21.563523    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-serving-cert: secret "maas-api-e2e-api-b-a813e2-serving-cert" not found
Oct 31 19:42:21.563667 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:42:21.563654    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls podName:74d29103-0cb9-4b84-8a7d-ff59d0da9e43 nodeName:}" failed. No retries permitted until 2026-10-31 19:42:22.063634437 +0000 UTC m=+1335.525009036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls") pod "maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" (UID: "74d29103-0cb9-4b84-8a7d-ff59d0da9e43") : secret "maas-api-e2e-api-b-a813e2-serving-cert" not found
Oct 31 19:42:21.563909 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.563890    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.566197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.566169    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:21.579419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:21.579393    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjgc\" (UniqueName: \"kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:22.177490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:22.177445    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:22.224650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:22.221352    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\") pod \"maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:22.240010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:22.239966    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:22.953663 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:42:22.953628    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d29103_0cb9_4b84_8a7d_ff59d0da9e43.slice/crio-d591503fca9c3ccfb9c7b0bcfaaafe42cc334a3e3f652d653956585ba242a8a7 WatchSource:0}: Error finding container d591503fca9c3ccfb9c7b0bcfaaafe42cc334a3e3f652d653956585ba242a8a7: Status 404 returned error can't find the container with id d591503fca9c3ccfb9c7b0bcfaaafe42cc334a3e3f652d653956585ba242a8a7
Oct 31 19:42:22.954095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:22.953719    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"]
Oct 31 19:42:23.629623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.629585    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"]
Oct 31 19:42:23.635179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.635156    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.641988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.641947    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b1fae7\""
Oct 31 19:42:23.645739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.644846    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b1fae7-dockercfg-wxnhl\""
Oct 31 19:42:23.651368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.651297    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"]
Oct 31 19:42:23.673793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.673752    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" event={"ID":"74d29103-0cb9-4b84-8a7d-ff59d0da9e43","Type":"ContainerStarted","Data":"51f02fed75861dbb07545fe29711f16665f0ad2b09b718fce14a38b01e9a5ebd"}
Oct 31 19:42:23.674005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.673819    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" event={"ID":"74d29103-0cb9-4b84-8a7d-ff59d0da9e43","Type":"ContainerStarted","Data":"d591503fca9c3ccfb9c7b0bcfaaafe42cc334a3e3f652d653956585ba242a8a7"}
Oct 31 19:42:23.674005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.673848    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:23.689019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.688985    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"]
Oct 31 19:42:23.695209 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.695185    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.703432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.703385    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.703756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.703731    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfln5\" (UniqueName: \"kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5\") pod \"payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.707115 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.706970    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"]
Oct 31 19:42:23.710944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.709444    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" podStartSLOduration=2.240274363 podStartE2EDuration="2.709433201s" totalImagesPullingTime="469.158838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:23.706698167 +0000 UTC m=+1337.168072785" watchObservedRunningTime="2026-10-31 19:42:23.709433201 +0000 UTC m=+1337.170807835"
Oct 31 19:42:23.805254 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.805221    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.805432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.805313    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfln5\" (UniqueName: \"kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5\") pod \"payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.805432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.805395    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh286\" (UniqueName: \"kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286\") pod \"payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.806163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.805969    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.806163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.806156    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.843110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.838630    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfln5\" (UniqueName: \"kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5\") pod \"payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:23.912567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.907966    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.912567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.908875    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.912567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.909187    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh286\" (UniqueName: \"kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286\") pod \"payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.946611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.946517    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh286\" (UniqueName: \"kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286\") pod \"payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:23.987570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:23.974060    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:24.026578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:24.024161    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:24.708381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:24.707040    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"]
Oct 31 19:42:24.710733 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:24.710624    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"]
Oct 31 19:42:24.732271 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:42:24.732242    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5825f6_65d5_4cf0_aea6_bb08007987e8.slice/crio-d021ffabb40450a9d76ff176031468540ab60398eceebf68bcb72e225806e947 WatchSource:0}: Error finding container d021ffabb40450a9d76ff176031468540ab60398eceebf68bcb72e225806e947: Status 404 returned error can't find the container with id d021ffabb40450a9d76ff176031468540ab60398eceebf68bcb72e225806e947
Oct 31 19:42:25.727168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.727126    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w" event={"ID":"41571f05-1414-4dbb-9f85-5505daae3307","Type":"ContainerStarted","Data":"10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb"}
Oct 31 19:42:25.727608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.727227    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w" event={"ID":"41571f05-1414-4dbb-9f85-5505daae3307","Type":"ContainerStarted","Data":"f949df8a55bd3170f4a48e19117fe14336379e54d15affe1033bd2d0799f45ce"}
Oct 31 19:42:25.727608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.727381    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:25.729019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.728992    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" event={"ID":"5d5825f6-65d5-4cf0-aea6-bb08007987e8","Type":"ContainerStarted","Data":"3b6bb0d11923e7a410828e0eca6e16fb62fc0dbdcacb247fbcc9d37ef86194bd"}
Oct 31 19:42:25.729150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.729036    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" event={"ID":"5d5825f6-65d5-4cf0-aea6-bb08007987e8","Type":"ContainerStarted","Data":"d021ffabb40450a9d76ff176031468540ab60398eceebf68bcb72e225806e947"}
Oct 31 19:42:25.729150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.729056    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:25.751898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.751846    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w" podStartSLOduration=2.751831529 podStartE2EDuration="2.751831529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:25.748155531 +0000 UTC m=+1339.209530147" watchObservedRunningTime="2026-10-31 19:42:25.751831529 +0000 UTC m=+1339.213206144"
Oct 31 19:42:25.772908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:25.772855    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" podStartSLOduration=2.77284218 podStartE2EDuration="2.77284218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:25.770882457 +0000 UTC m=+1339.232257074" watchObservedRunningTime="2026-10-31 19:42:25.77284218 +0000 UTC m=+1339.234216795"
Oct 31 19:42:29.722302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:29.722273    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:42:36.452853 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.452810    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"]
Oct 31 19:42:36.461450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.461414    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:36.467035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.466621    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"]
Oct 31 19:42:36.586845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.548884    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d45f6c\""
Oct 31 19:42:36.586845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.549036    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d45f6c-dockercfg-6lzks\""
Oct 31 19:42:36.606568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.598819    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9ln\" (UniqueName: \"kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln\") pod \"payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:36.646403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.646189    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:36.801683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.794430    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9ln\" (UniqueName: \"kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln\") pod \"payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:36.801683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.794683    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:36.863229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.862624    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:36.965304 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:36.965233    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9ln\" (UniqueName: \"kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln\") pod \"payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:37.079638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:37.079559    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:42:37.079811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:37.079650    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:42:37.154879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:37.154843    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:37.448692 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:42:37.448659    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aff6d5f_26da_423f_927d_37816b70747e.slice/crio-3489be8630888ebc9b75263289080db4deb9b017097555b708c25f82b06e7d74 WatchSource:0}: Error finding container 3489be8630888ebc9b75263289080db4deb9b017097555b708c25f82b06e7d74: Status 404 returned error can't find the container with id 3489be8630888ebc9b75263289080db4deb9b017097555b708c25f82b06e7d74
Oct 31 19:42:37.449144 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:37.449110    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"]
Oct 31 19:42:38.067379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:38.066170    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7" event={"ID":"1aff6d5f-26da-423f-927d-37816b70747e","Type":"ContainerStarted","Data":"cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f"}
Oct 31 19:42:38.067379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:38.066241    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7" event={"ID":"1aff6d5f-26da-423f-927d-37816b70747e","Type":"ContainerStarted","Data":"3489be8630888ebc9b75263289080db4deb9b017097555b708c25f82b06e7d74"}
Oct 31 19:42:38.067958 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:38.067597    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:38.175575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:38.169888    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7" podStartSLOduration=2.169864365 podStartE2EDuration="2.169864365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:38.095067345 +0000 UTC m=+1351.556441961" watchObservedRunningTime="2026-10-31 19:42:38.169864365 +0000 UTC m=+1351.631238982"
Oct 31 19:42:45.844577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:45.836804    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:42:45.919022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:45.908642    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:42:45.947635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:45.909851    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.079103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.079062    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.098056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.097951    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.301736 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.287672    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.314571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.313616    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.650570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.594044    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.750984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.750947    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-tbx78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:46.761612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.761562    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.810762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.806528    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.905192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.905090    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded"
Oct 31 19:42:46.920991 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.906025    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.988414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.922197    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.988414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.923714    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.988414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:46.948507    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.125217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.125171    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.271941 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.266243    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.317709    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.317766    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.317795    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.317822    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.317835    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"]
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.317879    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:42:47.343677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.319377    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy" containerID="cri-o://3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e" gracePeriod=30
Oct 31 19:42:47.388045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.362346    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 19:42:47.388045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.362726    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded"
Oct 31 19:42:47.389295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.389266    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:42:47.390871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.389496    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:42:47.398568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.394248    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:47.398568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.396748    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.465625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.465590    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 19:42:47.507714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.507680    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 19:42:47.603323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.585507    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"]
Oct 31 19:42:47.755620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.755526    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 19:42:47.763578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.756687    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 19:42:47.771581 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:42:47.701939    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:42:47.774569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.772055    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy" containerID="cri-o://b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df" gracePeriod=30
Oct 31 19:42:47.835692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.821400    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 19:42:47.841793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.841284    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 19:42:47.896857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:47.896828    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 19:42:49.606566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.569966    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"]
Oct 31 19:42:49.622666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.609190    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:49.742859 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.685207    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9efeae-dockercfg-9xwqg\""
Oct 31 19:42:49.742859 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.685426    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9efeae\""
Oct 31 19:42:49.811484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.808440    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254t8\" (UniqueName: \"kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8\") pod \"payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:49.821569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.814745    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:49.991601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:49.966155    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"]
Oct 31 19:42:50.115482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.115447    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:50.116238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.116210    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254t8\" (UniqueName: \"kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8\") pod \"payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:50.117793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.117766    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:50.160789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.160757    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"]
Oct 31 19:42:50.188565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.188519    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254t8\" (UniqueName: \"kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8\") pod \"payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:50.198837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.198752    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:42:50.198837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.198802    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"]
Oct 31 19:42:50.199954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.199930    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.321077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.321032    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.321276 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.321182    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgcv\" (UniqueName: \"kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv\") pod \"payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.423863 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.423822    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.424042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.423929    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgcv\" (UniqueName: \"kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv\") pod \"payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.424806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.424782    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.437676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.437642    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgcv\" (UniqueName: \"kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv\") pod \"payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.486015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.485938    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:50.549027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.548992    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:50.645600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.642330    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"]
Oct 31 19:42:50.650342 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.649618    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.668663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.667683    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b1fae7-serving-cert\""
Oct 31 19:42:50.688337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.688268    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"]
Oct 31 19:42:50.819731 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.817194    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"]
Oct 31 19:42:50.823506 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:42:50.823469    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfb7ff5_e74c_4b2a_b1b0_b3d954287adb.slice/crio-1aa764cf8a79804dac963a2e9b359a53bac0192e578ef5b467a3612fbe55e0c6 WatchSource:0}: Error finding container 1aa764cf8a79804dac963a2e9b359a53bac0192e578ef5b467a3612fbe55e0c6: Status 404 returned error can't find the container with id 1aa764cf8a79804dac963a2e9b359a53bac0192e578ef5b467a3612fbe55e0c6
Oct 31 19:42:50.835186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.835154    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwc8\" (UniqueName: \"kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.835352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.835328    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.835531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.835516    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.835625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.835580    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.943584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.941229    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwc8\" (UniqueName: \"kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.943584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.941369    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.943584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.941586    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:50.943584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.941652    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:51.005647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:50.953134    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:51.005647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.002137    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:51.006607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.006519    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwc8\" (UniqueName: \"kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:51.008976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.008898    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"]
Oct 31 19:42:51.016580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.015662    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls\") pod \"maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:51.041664 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:42:51.041628    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18a1d4c_40bd_4c67_a9ee_0a2955323ea9.slice/crio-26ffd2b4c62b92cda6915b32dacf2d5b042fec88e7d5796acab37f6d3152885b WatchSource:0}: Error finding container 26ffd2b4c62b92cda6915b32dacf2d5b042fec88e7d5796acab37f6d3152885b: Status 404 returned error can't find the container with id 26ffd2b4c62b92cda6915b32dacf2d5b042fec88e7d5796acab37f6d3152885b
Oct 31 19:42:51.197975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.197935    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" event={"ID":"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb","Type":"ContainerStarted","Data":"613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf"}
Oct 31 19:42:51.198171 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.198007    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" event={"ID":"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb","Type":"ContainerStarted","Data":"1aa764cf8a79804dac963a2e9b359a53bac0192e578ef5b467a3612fbe55e0c6"}
Oct 31 19:42:51.198699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.198676    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:42:51.215018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.203533    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" event={"ID":"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9","Type":"ContainerStarted","Data":"7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38"}
Oct 31 19:42:51.215018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.203602    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" event={"ID":"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9","Type":"ContainerStarted","Data":"26ffd2b4c62b92cda6915b32dacf2d5b042fec88e7d5796acab37f6d3152885b"}
Oct 31 19:42:51.215018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.204717    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:42:51.237061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.236992    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" podStartSLOduration=2.236972331 podStartE2EDuration="2.236972331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:51.230633124 +0000 UTC m=+1364.692007729" watchObservedRunningTime="2026-10-31 19:42:51.236972331 +0000 UTC m=+1364.698346947"
Oct 31 19:42:51.259665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.259604    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" podStartSLOduration=2.2595871 podStartE2EDuration="2.2595871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:51.256964337 +0000 UTC m=+1364.718338953" watchObservedRunningTime="2026-10-31 19:42:51.2595871 +0000 UTC m=+1364.720961714"
Oct 31 19:42:51.276944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.276899    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:51.591439 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:42:51.591404    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe49af0_be0b_4e31_b75b_5ca0ab985bee.slice/crio-419639c65bee4ecbd8bdb1417952adc5230c90d49ef9190f14c32deb7287e9d9 WatchSource:0}: Error finding container 419639c65bee4ecbd8bdb1417952adc5230c90d49ef9190f14c32deb7287e9d9: Status 404 returned error can't find the container with id 419639c65bee4ecbd8bdb1417952adc5230c90d49ef9190f14c32deb7287e9d9
Oct 31 19:42:51.592078 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:51.592052    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"]
Oct 31 19:42:52.240586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.240437    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" event={"ID":"cbe49af0-be0b-4e31-b75b-5ca0ab985bee","Type":"ContainerStarted","Data":"419639c65bee4ecbd8bdb1417952adc5230c90d49ef9190f14c32deb7287e9d9"}
Oct 31 19:42:52.720205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.720177    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:42:52.879225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879187    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.879225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879232    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879330    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879357    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879501    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879539    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879713    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879743    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.879795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.879746    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.880018    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.881831    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.882104    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.882177    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert\") pod \"3e8f67c6-c9e6-409b-9808-192635401cc5\" (UID: \"3e8f67c6-c9e6-409b-9808-192635401cc5\") "
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.882341    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.882683    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.883042    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.883054    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.883064    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.883073    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.883082    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8f67c6-c9e6-409b-9808-192635401cc5-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.886110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.883090    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.887846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.887809    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.888135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.888111    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "kube-api-access-rl6j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:52.889904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.889878    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token" pod "3e8f67c6-c9e6-409b-9808-192635401cc5" (UID: "3e8f67c6-c9e6-409b-9808-192635401cc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:52.985931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.985848    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.985931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.985881    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl6j5\" (UniqueName: \"kubernetes.io/projected/3e8f67c6-c9e6-409b-9808-192635401cc5-kube-api-access-rl6j5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.985931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:52.985896    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8f67c6-c9e6-409b-9808-192635401cc5-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.027037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.027002    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.217560    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.217633    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.217680    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.217752    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.217840    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.218020    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.218054    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.218090    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.218149    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket\") pod \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\" (UID: \"3b16dfa4-0e73-4df0-ac2d-a94961f993a8\") "
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.223859    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.224082    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.225890    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:53.237627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.226589    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:53.238613 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.238555    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:53.329571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.324813    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.329571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.324850    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.329571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.324862    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.329571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.324874    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.329571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.324886    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.349596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.344291    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:42:53.361222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.353679    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "kube-api-access-vwv4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:53.399873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.392765    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:53.399873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.395255    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" event={"ID":"cbe49af0-be0b-4e31-b75b-5ca0ab985bee","Type":"ContainerStarted","Data":"d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a"}
Oct 31 19:42:53.399873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.396620    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:42:53.399873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.399664    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy" pod "3b16dfa4-0e73-4df0-ac2d-a94961f993a8" (UID: "3b16dfa4-0e73-4df0-ac2d-a94961f993a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:53.526389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.521399    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" podStartSLOduration=3.05387616 podStartE2EDuration="3.521374473s" totalImagesPullingTime="467.498313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:53.461001817 +0000 UTC m=+1366.922376434" watchObservedRunningTime="2026-10-31 19:42:53.521374473 +0000 UTC m=+1366.982749088"
Oct 31 19:42:53.550620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.548115    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.550961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.550938    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.551249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.551231    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwv4x\" (UniqueName: \"kubernetes.io/projected/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-kube-api-access-vwv4x\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.551479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.551464    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b16dfa4-0e73-4df0-ac2d-a94961f993a8-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.559612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.559575    5958 generic.go:356] "Generic (PLEG): container finished" podID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerID="3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e" exitCode=0
Oct 31 19:42:53.560034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.559992    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" event={"ID":"3e8f67c6-c9e6-409b-9808-192635401cc5","Type":"ContainerDied","Data":"3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e"}
Oct 31 19:42:53.560272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.560237    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv" event={"ID":"3e8f67c6-c9e6-409b-9808-192635401cc5","Type":"ContainerDied","Data":"0535bfe4ad158976786f4ffa553a9ddd7ba1f999d2f50c5d1ef4e9736ee2674d"}
Oct 31 19:42:53.561510 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.561389    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"
Oct 31 19:42:53.561745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.561727    5958 scope.go:122] "RemoveContainer" containerID="3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e"
Oct 31 19:42:53.563626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.563603    5958 generic.go:356] "Generic (PLEG): container finished" podID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerID="b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df" exitCode=0
Oct 31 19:42:53.563888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.563866    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"
Oct 31 19:42:53.564723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.564700    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" event={"ID":"3b16dfa4-0e73-4df0-ac2d-a94961f993a8","Type":"ContainerDied","Data":"b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df"}
Oct 31 19:42:53.564830 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.564749    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp" event={"ID":"3b16dfa4-0e73-4df0-ac2d-a94961f993a8","Type":"ContainerDied","Data":"54938f22fe9d6d65165e56d38116184a75979e45183f9a87a70f3e139fcab745"}
Oct 31 19:42:53.618228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.610990    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"]
Oct 31 19:42:53.618228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.615788    5958 scope.go:122] "RemoveContainer" containerID="3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e"
Oct 31 19:42:53.618228 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:42:53.616210    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e\": container with ID starting with 3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e not found: ID does not exist" containerID="3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e"
Oct 31 19:42:53.618228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.616247    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e"} err="failed to get container status \"3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e\": rpc error: code = NotFound desc = could not find container \"3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e\": container with ID starting with 3da16bc0c03de1c8f3b71e894973abebddc2abb49e84c8f15b5e4e8b1093870e not found: ID does not exist"
Oct 31 19:42:53.618228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.616273    5958 scope.go:122] "RemoveContainer" containerID="b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df"
Oct 31 19:42:53.623250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.623223    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-gwppv"]
Oct 31 19:42:53.632581 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.632521    5958 scope.go:122] "RemoveContainer" containerID="b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df"
Oct 31 19:42:53.632949 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:42:53.632926    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df\": container with ID starting with b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df not found: ID does not exist" containerID="b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df"
Oct 31 19:42:53.633035 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.632963    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df"} err="failed to get container status \"b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df\": rpc error: code = NotFound desc = could not find container \"b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df\": container with ID starting with b5d9213331b10822e037a88486e3234e3f90224686d933d21f4cd23ecec892df not found: ID does not exist"
Oct 31 19:42:53.644655 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.644622    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"]
Oct 31 19:42:53.651945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:53.650068    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-wcgmp"]
Oct 31 19:42:54.706869 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:54.706833    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" path="/var/lib/kubelet/pods/3b16dfa4-0e73-4df0-ac2d-a94961f993a8/volumes"
Oct 31 19:42:54.707506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:42:54.707489    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" path="/var/lib/kubelet/pods/3e8f67c6-c9e6-409b-9808-192635401cc5/volumes"
Oct 31 19:43:00.591684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:00.591648    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:43:02.145238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.145204    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"]
Oct 31 19:43:02.146807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.146778    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy"
Oct 31 19:43:02.146807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.146809    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy"
Oct 31 19:43:02.147021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.146843    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy"
Oct 31 19:43:02.147021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.146852    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy"
Oct 31 19:43:02.147190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.147170    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b16dfa4-0e73-4df0-ac2d-a94961f993a8" containerName="istio-proxy"
Oct 31 19:43:02.147282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.147196    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8f67c6-c9e6-409b-9808-192635401cc5" containerName="istio-proxy"
Oct 31 19:43:02.154588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.154531    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.178927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.178897    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"]
Oct 31 19:43:02.216175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216096    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.216384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216213    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.216509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216379    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.216509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216476    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5n8\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.216654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216639    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.216924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216894    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.217016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.216979    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.217117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.217095    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.217229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.217212    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.241597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.241440    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-278c04-openshift-default-dockercfg-rsckc\""
Oct 31 19:43:02.244008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.243977    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.332850    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.332914    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.332985    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333091    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333157    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333210    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333294    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333394    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5n8\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333515    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333713    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.336576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.333951    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.355637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.345224    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.355637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.346389    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.355637 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.346688    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.428760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.428536    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.433213 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.433178    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.434563 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.434520    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5n8\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.439179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.439147    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.494974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.494927    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:02.799711 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:43:02.799675    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b1eb01_325e_4df9_9e5a_4ef67eb8c1c0.slice/crio-9750d42a68ddabd29ab04116685effa912adad001e3f23f65a4f6b8df2310924 WatchSource:0}: Error finding container 9750d42a68ddabd29ab04116685effa912adad001e3f23f65a4f6b8df2310924: Status 404 returned error can't find the container with id 9750d42a68ddabd29ab04116685effa912adad001e3f23f65a4f6b8df2310924
Oct 31 19:43:02.799927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.799906    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"]
Oct 31 19:43:02.802975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.802942    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:43:02.803077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.803005    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:43:02.803077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:02.803037    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:43:03.579839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:03.579799    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:43:03.793963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:03.793919    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" event={"ID":"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0","Type":"ContainerStarted","Data":"601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73"}
Oct 31 19:43:03.794150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:03.793981    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" event={"ID":"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0","Type":"ContainerStarted","Data":"9750d42a68ddabd29ab04116685effa912adad001e3f23f65a4f6b8df2310924"}
Oct 31 19:43:03.837894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:03.837669    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podStartSLOduration=1.837650167 podStartE2EDuration="1.837650167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:03.825440891 +0000 UTC m=+1377.286815507" watchObservedRunningTime="2026-10-31 19:43:03.837650167 +0000 UTC m=+1377.299024782"
Oct 31 19:43:04.495457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:04.495424    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:04.497105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:04.497077    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 19:43:04.497264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:04.497137    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 19:43:05.496009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:05.495965    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 19:43:05.496456 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:05.496031    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 19:43:06.495700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:06.495666    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 19:43:06.495913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:06.495755    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 19:43:07.497491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:07.497446    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 19:43:07.498057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:07.497529    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 19:43:08.496379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:08.496350    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 19:43:08.496605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:08.496583    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 19:43:09.499867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:09.496017    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 31 19:43:09.499867 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:09.496071    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 31 19:43:10.500837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:10.500804    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:10.501309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:10.501119    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:10.501923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:10.501904    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:43:16.486003 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.485963    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"]
Oct 31 19:43:16.502387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.502350    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.506592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.506565    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"]
Oct 31 19:43:16.509395 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.509366    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9efeae-serving-cert\""
Oct 31 19:43:16.632186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.632136    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.632809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.632769    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfnbk\" (UniqueName: \"kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.633130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.633111    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.633297 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.633275    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.787497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.783367    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfnbk\" (UniqueName: \"kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.787497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.783537    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.787497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.783611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.787497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.783725    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.787497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.784683    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.790134 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.790108    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.791765 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.791737    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.797645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.797619    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfnbk\" (UniqueName: \"kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk\") pod \"maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:16.828645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:16.828610    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:17.051844 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:43:17.051762    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:43:17.136942 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:43:17.136887    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14218bf_3190_4d4e_a0a5_ce9c2f9b032d.slice/crio-6a04ba5c7d9b58116c71087457627fb764bc6d7637481b2e728d1af9088fb1cb WatchSource:0}: Error finding container 6a04ba5c7d9b58116c71087457627fb764bc6d7637481b2e728d1af9088fb1cb: Status 404 returned error can't find the container with id 6a04ba5c7d9b58116c71087457627fb764bc6d7637481b2e728d1af9088fb1cb
Oct 31 19:43:17.137104 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:17.137075    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"]
Oct 31 19:43:17.332641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:17.329113    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj" event={"ID":"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d","Type":"ContainerStarted","Data":"6a04ba5c7d9b58116c71087457627fb764bc6d7637481b2e728d1af9088fb1cb"}
Oct 31 19:43:18.341324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:18.341288    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj" event={"ID":"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d","Type":"ContainerStarted","Data":"f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c"}
Oct 31 19:43:18.341828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:18.341392    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:18.365454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:18.365398    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj" podStartSLOduration=1.77629146 podStartE2EDuration="2.365378226s" totalImagesPullingTime="589.086766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:18.363226018 +0000 UTC m=+1391.824600629" watchObservedRunningTime="2026-10-31 19:43:18.365378226 +0000 UTC m=+1391.826752844"
Oct 31 19:43:24.562758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:24.562687    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:43:29.680419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:29.680355    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"]
Oct 31 19:43:29.686743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:29.686716    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:29.697044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:29.697009    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f8c3f7\""
Oct 31 19:43:29.697225 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:29.697202    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"]
Oct 31 19:43:29.933620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:29.863999    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:29.933620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:29.864256    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbj7\" (UniqueName: \"kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7\") pod \"payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:30.032976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:30.031783    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbj7\" (UniqueName: \"kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7\") pod \"payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:30.032976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:30.032068    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:30.116371 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:30.114194    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:30.263138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:30.261977    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbj7\" (UniqueName: \"kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7\") pod \"payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:30.342140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:30.342100    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:30.727332 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:30.727237    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"]
Oct 31 19:43:31.219786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:31.218168    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" event={"ID":"ddad558e-416e-460f-8d10-1c7b965efb7e","Type":"ContainerStarted","Data":"9581f5866e25ed7be70b11d47deebaf08f1b96dc2b921d4b5fc610818caad618"}
Oct 31 19:43:31.219786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:31.218265    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" event={"ID":"ddad558e-416e-460f-8d10-1c7b965efb7e","Type":"ContainerStarted","Data":"4a37f1d31d816fb04a8ecaebab1e14a1c09eac2d26579a8d83d978bcc86f925f"}
Oct 31 19:43:31.219786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:31.219688    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:31.277491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:31.277423    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" podStartSLOduration=2.2774023310000002 podStartE2EDuration="2.277402331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:31.257086285 +0000 UTC m=+1404.718460902" watchObservedRunningTime="2026-10-31 19:43:31.277402331 +0000 UTC m=+1404.738776947"
Oct 31 19:43:40.370535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:40.370490    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:43:43.303790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:43.303754    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:43:47.047867 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:43:47.047823    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:43:51.464723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:51.464670    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:43:52.957717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:52.957674    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"]
Oct 31 19:43:52.977566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:52.969225    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.022531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.005828    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f8c3f7-serving-cert\""
Oct 31 19:43:53.066467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.066426    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6zg\" (UniqueName: \"kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.082760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.082719    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.083055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.083033    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.083183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.083169    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.187563 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.187279    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"]
Oct 31 19:43:53.193805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.193725    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6zg\" (UniqueName: \"kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.193967 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.193853    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.193967 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.193901    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.193967 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.193933    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.194410 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:43:53.194377    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-serving-cert: secret "maas-api-e2e-inf-b-f8c3f7-serving-cert" not found
Oct 31 19:43:53.194483 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:43:53.194453    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls podName:53ea4bb4-066e-40d0-a94b-0acd3d0b91c3 nodeName:}" failed. No retries permitted until 2026-10-31 19:43:53.694431819 +0000 UTC m=+1427.155806430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls") pod "maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" (UID: "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3") : secret "maas-api-e2e-inf-b-f8c3f7-serving-cert" not found
Oct 31 19:43:53.197982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.195603    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.237307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.236568    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.288566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.287443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6zg\" (UniqueName: \"kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.709096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.708947    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:53.713359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:53.713331    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:54.016608 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:54.016156    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:54.463771 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:43:54.463739    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ea4bb4_066e_40d0_a94b_0acd3d0b91c3.slice/crio-35479bcec26305ae2661ddfe062e4da8928a91197c3b5e33a326de6284ed8105 WatchSource:0}: Error finding container 35479bcec26305ae2661ddfe062e4da8928a91197c3b5e33a326de6284ed8105: Status 404 returned error can't find the container with id 35479bcec26305ae2661ddfe062e4da8928a91197c3b5e33a326de6284ed8105
Oct 31 19:43:54.464183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:54.464157    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"]
Oct 31 19:43:55.040603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:55.040514    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" event={"ID":"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3","Type":"ContainerStarted","Data":"35479bcec26305ae2661ddfe062e4da8928a91197c3b5e33a326de6284ed8105"}
Oct 31 19:43:56.051573 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:56.051528    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" event={"ID":"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3","Type":"ContainerStarted","Data":"2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9"}
Oct 31 19:43:56.052344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:56.051762    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:43:56.075646 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:43:56.075584    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" podStartSLOduration=3.555349986 podStartE2EDuration="4.075566815s" totalImagesPullingTime="520.216829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:56.07528915 +0000 UTC m=+1429.536663789" watchObservedRunningTime="2026-10-31 19:43:56.075566815 +0000 UTC m=+1429.536941427"
Oct 31 19:44:02.154776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:02.154739    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:44:17.049042 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:44:17.048958    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:44:20.023982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:20.023941    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"]
Oct 31 19:44:20.024411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:20.024164    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy" containerID="cri-o://59fbbea1bfd54cd6e6a9482c4b34126776990b6e93f947803dc1d06718bb51c8" gracePeriod=30
Oct 31 19:44:25.417942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.417897    5958 generic.go:356] "Generic (PLEG): container finished" podID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerID="59fbbea1bfd54cd6e6a9482c4b34126776990b6e93f947803dc1d06718bb51c8" exitCode=0
Oct 31 19:44:25.418520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.418280    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" event={"ID":"c1fea17f-4800-4daa-b01a-a82b6dcf1201","Type":"ContainerDied","Data":"59fbbea1bfd54cd6e6a9482c4b34126776990b6e93f947803dc1d06718bb51c8"}
Oct 31 19:44:25.697720 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.697641    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:44:25.771444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771405    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.771661 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771602    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.771717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771657    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.771852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771827    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.771932 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771865    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.771988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.771988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.771970    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.772099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.772011    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.772099 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.772030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:25.772251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.772232    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs\") pod \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\" (UID: \"c1fea17f-4800-4daa-b01a-a82b6dcf1201\") "
Oct 31 19:44:25.772313 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.772256    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:25.772524 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.772501    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:25.772907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.772886    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:25.773344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.773321    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.773434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.773350    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.773434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.773364    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.773434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.773380    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.773834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.773811    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:25.775451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.775428    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "kube-api-access-jmdhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:25.775672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.775650    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:25.791669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.789945    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:44:25.791669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.790412    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token" pod "c1fea17f-4800-4daa-b01a-a82b6dcf1201" (UID: "c1fea17f-4800-4daa-b01a-a82b6dcf1201"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:25.886623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.882035    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdhq\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-kube-api-access-jmdhq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.886623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.882077    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.886623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.882092    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.886623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.882105    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fea17f-4800-4daa-b01a-a82b6dcf1201-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:25.886623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:25.882119    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fea17f-4800-4daa-b01a-a82b6dcf1201-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:26.428140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:26.428111    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"
Oct 31 19:44:26.428642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:26.428141    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq" event={"ID":"c1fea17f-4800-4daa-b01a-a82b6dcf1201","Type":"ContainerDied","Data":"f418f98a70fd1bf236e82e5285a5cf508b23f41b47e731176e75eedae4c321ca"}
Oct 31 19:44:26.428642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:26.428203    5958 scope.go:122] "RemoveContainer" containerID="59fbbea1bfd54cd6e6a9482c4b34126776990b6e93f947803dc1d06718bb51c8"
Oct 31 19:44:26.479579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:26.477453    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"]
Oct 31 19:44:26.500425 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:26.500390    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-cbxtq"]
Oct 31 19:44:26.727103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:26.726216    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" path="/var/lib/kubelet/pods/c1fea17f-4800-4daa-b01a-a82b6dcf1201/volumes"
Oct 31 19:44:38.820668 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:44:38.820630    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:44:47.048460 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:44:47.048414    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:45:00.051124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.051079    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"]
Oct 31 19:45:00.052783 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.052757    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy"
Oct 31 19:45:00.052783 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.052784    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy"
Oct 31 19:45:00.053010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.052998    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fea17f-4800-4daa-b01a-a82b6dcf1201" containerName="istio-proxy"
Oct 31 19:45:00.064025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.063980    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"]
Oct 31 19:45:00.064252 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.064109    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.077435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.077401    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d8c398\""
Oct 31 19:45:00.171569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.137204    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d8c398-dockercfg-zv2hq\""
Oct 31 19:45:00.582826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.582753    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d8c398-546ff884f-pr754\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.583028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.582924    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbbn\" (UniqueName: \"kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn\") pod \"payload-processing-e2e-inf-b-d8c398-546ff884f-pr754\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.653837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.633305    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"]
Oct 31 19:45:00.669255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.668656    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.669661 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.669638    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"]
Oct 31 19:45:00.681827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.681792    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:45:00.682430 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.682406    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"]
Oct 31 19:45:00.682821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.682801    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.684413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.684382    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d8c398-546ff884f-pr754\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.684521 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.684496    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbbn\" (UniqueName: \"kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn\") pod \"payload-processing-e2e-inf-b-d8c398-546ff884f-pr754\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.685466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.685445    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d8c398-546ff884f-pr754\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.690114 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.690092    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"]
Oct 31 19:45:00.690995 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.690406    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.697514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.697438    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"]
Oct 31 19:45:00.697694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.697623    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.698157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.698132    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbbn\" (UniqueName: \"kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn\") pod \"payload-processing-e2e-inf-b-d8c398-546ff884f-pr754\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.703151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.703125    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"]
Oct 31 19:45:00.703836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.703813    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.725920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.722739    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"]
Oct 31 19:45:00.725920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.722860    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:00.736057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.736018    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:00.752840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.752807    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"]
Oct 31 19:45:00.752995 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.752872    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"]
Oct 31 19:45:00.759834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.759807    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"]
Oct 31 19:45:00.760029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.759856    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"]
Oct 31 19:45:00.760029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.759884    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"]
Oct 31 19:45:00.760029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.759966    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:00.766499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.766371    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"]
Oct 31 19:45:00.766499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.766418    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"]
Oct 31 19:45:00.766499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.766446    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"]
Oct 31 19:45:00.767000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.766555    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:00.771855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.771825    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"]
Oct 31 19:45:00.778486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.778460    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"]
Oct 31 19:45:00.786630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.784867    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"]
Oct 31 19:45:00.786630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786181    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhdxj\" (UniqueName: \"kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj\") pod \"maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.786630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786345    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48k4f\" (UniqueName: \"kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.786630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786448    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.786630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786579    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.787016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786660    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wp6\" (UniqueName: \"kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6\") pod \"maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.787016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786687    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.787016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786876    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.787016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.786934    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzdq\" (UniqueName: \"kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.787238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.787035    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbl2\" (UniqueName: \"kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.787238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.787125    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.889125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889084    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.889125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889133    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wp6\" (UniqueName: \"kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6\") pod \"maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.889410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889255    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.889410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889391    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthv7\" (UniqueName: \"kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7\") pod \"maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:00.889520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889443    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.889614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889563    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:00.889675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889622    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzdq\" (UniqueName: \"kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.889769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889752    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9szk\" (UniqueName: \"kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk\") pod \"maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:00.889821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889809    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbl2\" (UniqueName: \"kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.889943 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889921    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.890141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.889976    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.890270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890172    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.890270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890191    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhdxj\" (UniqueName: \"kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj\") pod \"maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.890270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890065    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.890270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890231    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2x5q\" (UniqueName: \"kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q\") pod \"maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:00.890501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890308    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48k4f\" (UniqueName: \"kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.890878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890794    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.890972 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.890918    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:00.891125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.891100    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8272h\" (UniqueName: \"kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h\") pod \"maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:00.891250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.891167    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.891250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.891196    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:00.891370 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.891269    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:00.891960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.891937    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.907129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.907089    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wp6\" (UniqueName: \"kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6\") pod \"maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:00.907129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.907105    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzdq\" (UniqueName: \"kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:00.907528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.907501    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48k4f\" (UniqueName: \"kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:00.908707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.908684    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbl2\" (UniqueName: \"kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:00.910381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.910357    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhdxj\" (UniqueName: \"kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj\") pod \"maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:00.988799 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.988764    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:00.992686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.992659    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:00.992825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.992731    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:00.992897 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.992880    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthv7\" (UniqueName: \"kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7\") pod \"maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:00.992970 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.992939    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:00.993045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993031    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9szk\" (UniqueName: \"kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk\") pod \"maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:00.993644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993169    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2x5q\" (UniqueName: \"kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q\") pod \"maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:00.993644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993253    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:00.993644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993294    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8272h\" (UniqueName: \"kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h\") pod \"maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:00.993644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993420    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:00.993644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993460    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:00.993841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.993713    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:00.994087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:00.994062    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:01.022575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.020888    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:01.062814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.040276    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9szk\" (UniqueName: \"kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk\") pod \"maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:01.062814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.060860    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:01.070016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.069985    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:01.083611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.076207    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8272h\" (UniqueName: \"kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h\") pod \"maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:01.083611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.078834    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:01.092833 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.088221    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthv7\" (UniqueName: \"kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7\") pod \"maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:01.096856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.096796    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:01.112215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.111526    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2x5q\" (UniqueName: \"kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q\") pod \"maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:01.112215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.111985    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:01.139999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.139760    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:01.152186 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.151907    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:01.160678 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.159226    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:01.662904 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:01.662868    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed33b55_7692_4bbc_bcd0_82522ef33dab.slice/crio-4d2f8aeed2f7837540ed46146d6f145ad61f980de46c98e614bef4ca269bd152 WatchSource:0}: Error finding container 4d2f8aeed2f7837540ed46146d6f145ad61f980de46c98e614bef4ca269bd152: Status 404 returned error can't find the container with id 4d2f8aeed2f7837540ed46146d6f145ad61f980de46c98e614bef4ca269bd152
Oct 31 19:45:01.663811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.663685    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"]
Oct 31 19:45:01.708506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.708462    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"]
Oct 31 19:45:01.723187 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.722915    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"]
Oct 31 19:45:01.731043 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:01.728299    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc564e69_d622_4aec_b65a_5ae8c86682b8.slice/crio-55b075d09205b77a73bc79fdc79dce495c6b07cc9d4cd1e5374d641c7aa5944e WatchSource:0}: Error finding container 55b075d09205b77a73bc79fdc79dce495c6b07cc9d4cd1e5374d641c7aa5944e: Status 404 returned error can't find the container with id 55b075d09205b77a73bc79fdc79dce495c6b07cc9d4cd1e5374d641c7aa5944e
Oct 31 19:45:01.873285 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:01.873251    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432b825c_daf9_4e62_aa2e_9010e92b761f.slice/crio-9eeebb886d662904f853fff194d68dd3983b9f04be849b2daf705464962502a9 WatchSource:0}: Error finding container 9eeebb886d662904f853fff194d68dd3983b9f04be849b2daf705464962502a9: Status 404 returned error can't find the container with id 9eeebb886d662904f853fff194d68dd3983b9f04be849b2daf705464962502a9
Oct 31 19:45:01.874343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.874005    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"]
Oct 31 19:45:01.913649 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.906694    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"]
Oct 31 19:45:01.923789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.921342    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"]
Oct 31 19:45:01.930984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.930953    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"]
Oct 31 19:45:01.938030 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.937864    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"]
Oct 31 19:45:01.947654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.947592    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"]
Oct 31 19:45:01.957731 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:01.953633    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"]
Oct 31 19:45:01.963389 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:01.963356    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a09253_00e3_4a3f_8720_d8ad07fc2d85.slice/crio-16a19336bf760606e96eae9b9632c10abc207491315be5ab68e843e82638d776 WatchSource:0}: Error finding container 16a19336bf760606e96eae9b9632c10abc207491315be5ab68e843e82638d776: Status 404 returned error can't find the container with id 16a19336bf760606e96eae9b9632c10abc207491315be5ab68e843e82638d776
Oct 31 19:45:02.127202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.127170    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerStarted","Data":"9eeebb886d662904f853fff194d68dd3983b9f04be849b2daf705464962502a9"}
Oct 31 19:45:02.128869 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.128839    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerStarted","Data":"2c2fa151fe92921f868ad7d29d56f92ee222d8fb65dc075e26a9efbeab007127"}
Oct 31 19:45:02.130231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.130197    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerStarted","Data":"67a4d2b4b6b9a914a474883276a06815d5cd4ead3d6b576e84cd1984dce14fb6"}
Oct 31 19:45:02.131598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.131573    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" event={"ID":"02074908-0ac2-49f0-a639-69dc35a0b5f6","Type":"ContainerStarted","Data":"4b218ae40923c528c530e84e483ada8ec6e657a33cc6864d0771271cfd1ac077"}
Oct 31 19:45:02.133586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.133537    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754" event={"ID":"2ed33b55-7692-4bbc-bcd0-82522ef33dab","Type":"ContainerStarted","Data":"cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302"}
Oct 31 19:45:02.133677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.133612    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:02.133677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.133637    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754" event={"ID":"2ed33b55-7692-4bbc-bcd0-82522ef33dab","Type":"ContainerStarted","Data":"4d2f8aeed2f7837540ed46146d6f145ad61f980de46c98e614bef4ca269bd152"}
Oct 31 19:45:02.135042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.135018    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerStarted","Data":"36c884dcb15717ee1fc8ada517a9936979d954db46ad1d684ccc4844e852550d"}
Oct 31 19:45:02.136365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.136338    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" event={"ID":"05a09253-00e3-4a3f-8720-d8ad07fc2d85","Type":"ContainerStarted","Data":"16a19336bf760606e96eae9b9632c10abc207491315be5ab68e843e82638d776"}
Oct 31 19:45:02.137633 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.137609    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerStarted","Data":"9ef0fc00c515c243b555882b2df698d3df8102c300a66c25e526890666c53219"}
Oct 31 19:45:02.139150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.139119    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerStarted","Data":"56a8f8db8839ffc772b809863b2c042ca7e90994cd61278c3da65a5ce62bfb9c"}
Oct 31 19:45:02.140585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.140557    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerStarted","Data":"55b075d09205b77a73bc79fdc79dce495c6b07cc9d4cd1e5374d641c7aa5944e"}
Oct 31 19:45:02.157788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:02.157727    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754" podStartSLOduration=2.1577137889999998 podStartE2EDuration="2.157713789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:02.155686074 +0000 UTC m=+1495.617060689" watchObservedRunningTime="2026-10-31 19:45:02.157713789 +0000 UTC m=+1495.619088403"
Oct 31 19:45:03.254207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.244900    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerStarted","Data":"35d8620f83b824eff0f82fef7f837c307ae1eed9a785d2e89aced56ae6ebafcd"}
Oct 31 19:45:03.270480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.270438    5958 scope.go:122] "RemoveContainer" containerID="26d8ada6b4d21342e9121f7361d10a814b0b2fa9891d269d81c8aa4608fd5c3a"
Oct 31 19:45:03.271314 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.271285    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerStarted","Data":"26d8ada6b4d21342e9121f7361d10a814b0b2fa9891d269d81c8aa4608fd5c3a"}
Oct 31 19:45:03.279776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.279273    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerStarted","Data":"2889076d728fb10f9fbcfb213b900e039defd6c37c947640a9c9a80d83522e4b"}
Oct 31 19:45:03.279776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.279650    5958 scope.go:122] "RemoveContainer" containerID="2889076d728fb10f9fbcfb213b900e039defd6c37c947640a9c9a80d83522e4b"
Oct 31 19:45:03.283655 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.283199    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerStarted","Data":"ee1346d46dfe303b8df9827a53f7dcae1c8d3a12aa18c4b79108c1aa948a829e"}
Oct 31 19:45:03.287072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.285416    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" podStartSLOduration=3.2853990189999998 podStartE2EDuration="3.285399019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:03.277283393 +0000 UTC m=+1496.738658008" watchObservedRunningTime="2026-10-31 19:45:03.285399019 +0000 UTC m=+1496.746773635"
Oct 31 19:45:03.287072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.285658    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerStarted","Data":"007863a111b273744e6f6646614a19fc6d5a31ca7f1a6e2f98e651e4b8954714"}
Oct 31 19:45:03.288233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.288124    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerStarted","Data":"60ea450afe2e1413a18ce6fc1dd8f21cd554eab72b6bca13a254b3ddfef75ff0"}
Oct 31 19:45:03.289568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.288672    5958 scope.go:122] "RemoveContainer" containerID="60ea450afe2e1413a18ce6fc1dd8f21cd554eab72b6bca13a254b3ddfef75ff0"
Oct 31 19:45:03.305656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.304467    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerStarted","Data":"01e08de0dedc911ad4ecf92989e983417d0065292af83dee76fc13f22b93e094"}
Oct 31 19:45:03.331652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.331531    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" podStartSLOduration=3.331506942 podStartE2EDuration="3.331506942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:03.322594242 +0000 UTC m=+1496.783968847" watchObservedRunningTime="2026-10-31 19:45:03.331506942 +0000 UTC m=+1496.792881559"
Oct 31 19:45:03.357567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.357457    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" podStartSLOduration=3.357434374 podStartE2EDuration="3.357434374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:03.34945361 +0000 UTC m=+1496.810828225" watchObservedRunningTime="2026-10-31 19:45:03.357434374 +0000 UTC m=+1496.818808990"
Oct 31 19:45:03.421042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.420990    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" podStartSLOduration=3.420975994 podStartE2EDuration="3.420975994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:03.416002078 +0000 UTC m=+1496.877376695" watchObservedRunningTime="2026-10-31 19:45:03.420975994 +0000 UTC m=+1496.882350625"
Oct 31 19:45:03.437421 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:03.437368    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" podStartSLOduration=3.437353524 podStartE2EDuration="3.437353524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:03.435701298 +0000 UTC m=+1496.897075914" watchObservedRunningTime="2026-10-31 19:45:03.437353524 +0000 UTC m=+1496.898728140"
Oct 31 19:45:04.315276 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.315241    5958 generic.go:356] "Generic (PLEG): container finished" podID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerID="ee1346d46dfe303b8df9827a53f7dcae1c8d3a12aa18c4b79108c1aa948a829e" exitCode=60
Oct 31 19:45:04.315721 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.315279    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerDied","Data":"ee1346d46dfe303b8df9827a53f7dcae1c8d3a12aa18c4b79108c1aa948a829e"}
Oct 31 19:45:04.315844 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.315823    5958 scope.go:122] "RemoveContainer" containerID="ee1346d46dfe303b8df9827a53f7dcae1c8d3a12aa18c4b79108c1aa948a829e"
Oct 31 19:45:04.321864 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.321836    5958 generic.go:356] "Generic (PLEG): container finished" podID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerID="007863a111b273744e6f6646614a19fc6d5a31ca7f1a6e2f98e651e4b8954714" exitCode=60
Oct 31 19:45:04.321970 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.321922    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerDied","Data":"007863a111b273744e6f6646614a19fc6d5a31ca7f1a6e2f98e651e4b8954714"}
Oct 31 19:45:04.322308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.322280    5958 scope.go:122] "RemoveContainer" containerID="007863a111b273744e6f6646614a19fc6d5a31ca7f1a6e2f98e651e4b8954714"
Oct 31 19:45:04.325333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.325307    5958 generic.go:356] "Generic (PLEG): container finished" podID="80029600-68bc-428c-adda-705e0b073238" containerID="60ea450afe2e1413a18ce6fc1dd8f21cd554eab72b6bca13a254b3ddfef75ff0" exitCode=60
Oct 31 19:45:04.325432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.325412    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerDied","Data":"60ea450afe2e1413a18ce6fc1dd8f21cd554eab72b6bca13a254b3ddfef75ff0"}
Oct 31 19:45:04.325479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.325460    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerStarted","Data":"4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648"}
Oct 31 19:45:04.328343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.328310    5958 generic.go:356] "Generic (PLEG): container finished" podID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerID="e0e7b00f4ef1817d8e730d0b6e89777580fb622f9925984c56cd94f8d6d82b7c" exitCode=60
Oct 31 19:45:04.328450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.328411    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" event={"ID":"02074908-0ac2-49f0-a639-69dc35a0b5f6","Type":"ContainerDied","Data":"e0e7b00f4ef1817d8e730d0b6e89777580fb622f9925984c56cd94f8d6d82b7c"}
Oct 31 19:45:04.328924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.328903    5958 scope.go:122] "RemoveContainer" containerID="e0e7b00f4ef1817d8e730d0b6e89777580fb622f9925984c56cd94f8d6d82b7c"
Oct 31 19:45:04.335097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.334172    5958 generic.go:356] "Generic (PLEG): container finished" podID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerID="01e08de0dedc911ad4ecf92989e983417d0065292af83dee76fc13f22b93e094" exitCode=60
Oct 31 19:45:04.335097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.334323    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerDied","Data":"01e08de0dedc911ad4ecf92989e983417d0065292af83dee76fc13f22b93e094"}
Oct 31 19:45:04.335097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.334782    5958 scope.go:122] "RemoveContainer" containerID="01e08de0dedc911ad4ecf92989e983417d0065292af83dee76fc13f22b93e094"
Oct 31 19:45:04.348884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.347820    5958 generic.go:356] "Generic (PLEG): container finished" podID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerID="ce6c2d4738edec7de4623ef75665a9bbce6f8311553bf4b88ffaed0eb9e34269" exitCode=60
Oct 31 19:45:04.348884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.347990    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" event={"ID":"05a09253-00e3-4a3f-8720-d8ad07fc2d85","Type":"ContainerDied","Data":"ce6c2d4738edec7de4623ef75665a9bbce6f8311553bf4b88ffaed0eb9e34269"}
Oct 31 19:45:04.348884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.348484    5958 scope.go:122] "RemoveContainer" containerID="ce6c2d4738edec7de4623ef75665a9bbce6f8311553bf4b88ffaed0eb9e34269"
Oct 31 19:45:04.367839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.367797    5958 generic.go:356] "Generic (PLEG): container finished" podID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerID="35d8620f83b824eff0f82fef7f837c307ae1eed9a785d2e89aced56ae6ebafcd" exitCode=60
Oct 31 19:45:04.368038 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.367927    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerDied","Data":"35d8620f83b824eff0f82fef7f837c307ae1eed9a785d2e89aced56ae6ebafcd"}
Oct 31 19:45:04.372891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.368465    5958 scope.go:122] "RemoveContainer" containerID="35d8620f83b824eff0f82fef7f837c307ae1eed9a785d2e89aced56ae6ebafcd"
Oct 31 19:45:04.387776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.385208    5958 generic.go:356] "Generic (PLEG): container finished" podID="1464acad-4fff-4e70-918c-5c146d76c775" containerID="26d8ada6b4d21342e9121f7361d10a814b0b2fa9891d269d81c8aa4608fd5c3a" exitCode=60
Oct 31 19:45:04.387776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.385654    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerDied","Data":"26d8ada6b4d21342e9121f7361d10a814b0b2fa9891d269d81c8aa4608fd5c3a"}
Oct 31 19:45:04.387776 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.385709    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerStarted","Data":"12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd"}
Oct 31 19:45:04.394066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.393098    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerID="2889076d728fb10f9fbcfb213b900e039defd6c37c947640a9c9a80d83522e4b" exitCode=60
Oct 31 19:45:04.394066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.393159    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerDied","Data":"2889076d728fb10f9fbcfb213b900e039defd6c37c947640a9c9a80d83522e4b"}
Oct 31 19:45:04.394066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.393192    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerStarted","Data":"22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332"}
Oct 31 19:45:04.426203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.426145    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" podStartSLOduration=4.426129037 podStartE2EDuration="4.426129037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.422984037 +0000 UTC m=+1497.884358666" watchObservedRunningTime="2026-10-31 19:45:04.426129037 +0000 UTC m=+1497.887503651"
Oct 31 19:45:04.469648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.469503    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" podStartSLOduration=4.469488695 podStartE2EDuration="4.469488695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.46542057 +0000 UTC m=+1497.926795186" watchObservedRunningTime="2026-10-31 19:45:04.469488695 +0000 UTC m=+1497.930863378"
Oct 31 19:45:04.494368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:04.494317    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" podStartSLOduration=4.494301621 podStartE2EDuration="4.494301621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.488803038 +0000 UTC m=+1497.950177659" watchObservedRunningTime="2026-10-31 19:45:04.494301621 +0000 UTC m=+1497.955676236"
Oct 31 19:45:05.405121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.405088    5958 generic.go:356] "Generic (PLEG): container finished" podID="80029600-68bc-428c-adda-705e0b073238" containerID="4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648" exitCode=60
Oct 31 19:45:05.405761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.405735    5958 scope.go:122] "RemoveContainer" containerID="4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648"
Oct 31 19:45:05.406184 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:05.406157    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb_odh-ai-gateway-infra(80029600-68bc-428c-adda-705e0b073238)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" podUID="80029600-68bc-428c-adda-705e0b073238"
Oct 31 19:45:05.406596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.406539    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerDied","Data":"4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648"}
Oct 31 19:45:05.406727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.406635    5958 scope.go:122] "RemoveContainer" containerID="60ea450afe2e1413a18ce6fc1dd8f21cd554eab72b6bca13a254b3ddfef75ff0"
Oct 31 19:45:05.411285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.411255    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" event={"ID":"02074908-0ac2-49f0-a639-69dc35a0b5f6","Type":"ContainerStarted","Data":"690902cb04bff94a860ad1313160c87042759a44dde2cc59efa123bacbd6bf70"}
Oct 31 19:45:05.413879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.413827    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerStarted","Data":"92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd"}
Oct 31 19:45:05.416217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.416192    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" event={"ID":"05a09253-00e3-4a3f-8720-d8ad07fc2d85","Type":"ContainerStarted","Data":"531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f"}
Oct 31 19:45:05.418399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.418377    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerStarted","Data":"33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17"}
Oct 31 19:45:05.420461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.420435    5958 generic.go:356] "Generic (PLEG): container finished" podID="1464acad-4fff-4e70-918c-5c146d76c775" containerID="12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd" exitCode=60
Oct 31 19:45:05.420627 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.420605    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerDied","Data":"12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd"}
Oct 31 19:45:05.420849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.420831    5958 scope.go:122] "RemoveContainer" containerID="12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd"
Oct 31 19:45:05.421218 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:05.421194    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x_odh-ai-gateway-infra(1464acad-4fff-4e70-918c-5c146d76c775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" podUID="1464acad-4fff-4e70-918c-5c146d76c775"
Oct 31 19:45:05.422757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.422734    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerID="22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332" exitCode=60
Oct 31 19:45:05.422851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.422817    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerDied","Data":"22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332"}
Oct 31 19:45:05.423043 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.423028    5958 scope.go:122] "RemoveContainer" containerID="22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332"
Oct 31 19:45:05.423385 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:05.423360    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw_odh-ai-gateway-infra(fc564e69-d622-4aec-b65a-5ae8c86682b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8"
Oct 31 19:45:05.424936 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.424841    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerStarted","Data":"79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1"}
Oct 31 19:45:05.424989 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.424942    5958 scope.go:122] "RemoveContainer" containerID="26d8ada6b4d21342e9121f7361d10a814b0b2fa9891d269d81c8aa4608fd5c3a"
Oct 31 19:45:05.426836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.426815    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerStarted","Data":"235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4"}
Oct 31 19:45:05.454739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.454710    5958 scope.go:122] "RemoveContainer" containerID="2889076d728fb10f9fbcfb213b900e039defd6c37c947640a9c9a80d83522e4b"
Oct 31 19:45:05.536097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:05.536042    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" podStartSLOduration=5.536023302 podStartE2EDuration="5.536023302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:05.510770714 +0000 UTC m=+1498.972145331" watchObservedRunningTime="2026-10-31 19:45:05.536023302 +0000 UTC m=+1498.997397918"
Oct 31 19:45:06.439315 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.439275    5958 generic.go:356] "Generic (PLEG): container finished" podID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerID="235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4" exitCode=60
Oct 31 19:45:06.439850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.439321    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerDied","Data":"235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4"}
Oct 31 19:45:06.439850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.439382    5958 scope.go:122] "RemoveContainer" containerID="007863a111b273744e6f6646614a19fc6d5a31ca7f1a6e2f98e651e4b8954714"
Oct 31 19:45:06.439850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.439815    5958 scope.go:122] "RemoveContainer" containerID="235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4"
Oct 31 19:45:06.440271 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.440246    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn_odh-ai-gateway-infra(65f3803a-fe05-4b98-9c24-c15cb6291361)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361"
Oct 31 19:45:06.442489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.442467    5958 scope.go:122] "RemoveContainer" containerID="4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648"
Oct 31 19:45:06.442920 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.442899    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb_odh-ai-gateway-infra(80029600-68bc-428c-adda-705e0b073238)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" podUID="80029600-68bc-428c-adda-705e0b073238"
Oct 31 19:45:06.444432 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.444403    5958 generic.go:356] "Generic (PLEG): container finished" podID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerID="690902cb04bff94a860ad1313160c87042759a44dde2cc59efa123bacbd6bf70" exitCode=60
Oct 31 19:45:06.444638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.444588    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" event={"ID":"02074908-0ac2-49f0-a639-69dc35a0b5f6","Type":"ContainerDied","Data":"690902cb04bff94a860ad1313160c87042759a44dde2cc59efa123bacbd6bf70"}
Oct 31 19:45:06.445122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.445100    5958 scope.go:122] "RemoveContainer" containerID="690902cb04bff94a860ad1313160c87042759a44dde2cc59efa123bacbd6bf70"
Oct 31 19:45:06.445607 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.445577    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm_odh-ai-gateway-infra(02074908-0ac2-49f0-a639-69dc35a0b5f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6"
Oct 31 19:45:06.446948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.446917    5958 generic.go:356] "Generic (PLEG): container finished" podID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerID="92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd" exitCode=60
Oct 31 19:45:06.447056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.447001    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerDied","Data":"92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd"}
Oct 31 19:45:06.447290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.447274    5958 scope.go:122] "RemoveContainer" containerID="92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd"
Oct 31 19:45:06.447669 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.447647    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh_odh-ai-gateway-infra(839dd42e-ea6a-47e4-b673-ba22730ce33e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e"
Oct 31 19:45:06.449196 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.449177    5958 generic.go:356] "Generic (PLEG): container finished" podID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerID="531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f" exitCode=60
Oct 31 19:45:06.449290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.449257    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" event={"ID":"05a09253-00e3-4a3f-8720-d8ad07fc2d85","Type":"ContainerDied","Data":"531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f"}
Oct 31 19:45:06.449531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.449516    5958 scope.go:122] "RemoveContainer" containerID="531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f"
Oct 31 19:45:06.449901 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.449879    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8_odh-ai-gateway-infra(05a09253-00e3-4a3f-8720-d8ad07fc2d85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85"
Oct 31 19:45:06.451182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.451160    5958 generic.go:356] "Generic (PLEG): container finished" podID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerID="33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17" exitCode=60
Oct 31 19:45:06.451315 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.451241    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerDied","Data":"33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17"}
Oct 31 19:45:06.451472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.451454    5958 scope.go:122] "RemoveContainer" containerID="33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17"
Oct 31 19:45:06.451913 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.451894    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch_odh-ai-gateway-infra(367ea9de-5a5c-4006-9bef-80e040470bf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2"
Oct 31 19:45:06.453472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.453450    5958 scope.go:122] "RemoveContainer" containerID="12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd"
Oct 31 19:45:06.453827 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.453804    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x_odh-ai-gateway-infra(1464acad-4fff-4e70-918c-5c146d76c775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" podUID="1464acad-4fff-4e70-918c-5c146d76c775"
Oct 31 19:45:06.455271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.455256    5958 scope.go:122] "RemoveContainer" containerID="22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332"
Oct 31 19:45:06.455602 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.455578    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw_odh-ai-gateway-infra(fc564e69-d622-4aec-b65a-5ae8c86682b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8"
Oct 31 19:45:06.457109 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.457048    5958 generic.go:356] "Generic (PLEG): container finished" podID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerID="79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1" exitCode=60
Oct 31 19:45:06.457185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.457115    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerDied","Data":"79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1"}
Oct 31 19:45:06.457185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.457142    5958 scope.go:122] "RemoveContainer" containerID="e0e7b00f4ef1817d8e730d0b6e89777580fb622f9925984c56cd94f8d6d82b7c"
Oct 31 19:45:06.457411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.457392    5958 scope.go:122] "RemoveContainer" containerID="79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1"
Oct 31 19:45:06.457693 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:06.457672    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg_odh-ai-gateway-infra(432b825c-daf9-4e62-aa2e-9010e92b761f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f"
Oct 31 19:45:06.475723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.475689    5958 scope.go:122] "RemoveContainer" containerID="01e08de0dedc911ad4ecf92989e983417d0065292af83dee76fc13f22b93e094"
Oct 31 19:45:06.494091 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.494071    5958 scope.go:122] "RemoveContainer" containerID="ce6c2d4738edec7de4623ef75665a9bbce6f8311553bf4b88ffaed0eb9e34269"
Oct 31 19:45:06.510104 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.510087    5958 scope.go:122] "RemoveContainer" containerID="35d8620f83b824eff0f82fef7f837c307ae1eed9a785d2e89aced56ae6ebafcd"
Oct 31 19:45:06.528845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:06.528821    5958 scope.go:122] "RemoveContainer" containerID="ee1346d46dfe303b8df9827a53f7dcae1c8d3a12aa18c4b79108c1aa948a829e"
Oct 31 19:45:07.470079 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:07.470049    5958 scope.go:122] "RemoveContainer" containerID="92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd"
Oct 31 19:45:07.470599 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:07.470532    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh_odh-ai-gateway-infra(839dd42e-ea6a-47e4-b673-ba22730ce33e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e"
Oct 31 19:45:07.479305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:07.479279    5958 scope.go:122] "RemoveContainer" containerID="531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f"
Oct 31 19:45:07.479790 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:07.479754    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8_odh-ai-gateway-infra(05a09253-00e3-4a3f-8720-d8ad07fc2d85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85"
Oct 31 19:45:07.481536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:07.481501    5958 scope.go:122] "RemoveContainer" containerID="33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17"
Oct 31 19:45:07.481966 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:07.481945    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch_odh-ai-gateway-infra(367ea9de-5a5c-4006-9bef-80e040470bf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2"
Oct 31 19:45:07.483825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:07.483802    5958 scope.go:122] "RemoveContainer" containerID="79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1"
Oct 31 19:45:07.484233 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:07.484204    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg_odh-ai-gateway-infra(432b825c-daf9-4e62-aa2e-9010e92b761f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f"
Oct 31 19:45:07.485379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:07.485348    5958 scope.go:122] "RemoveContainer" containerID="235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4"
Oct 31 19:45:07.485720 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:07.485701    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn_odh-ai-gateway-infra(65f3803a-fe05-4b98-9c24-c15cb6291361)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361"
Oct 31 19:45:09.238568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:09.223635    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"]
Oct 31 19:45:09.238568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:09.223892    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" podUID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" containerName="maas-api" containerID="cri-o://51f02fed75861dbb07545fe29711f16665f0ad2b09b718fce14a38b01e9a5ebd" gracePeriod=30
Oct 31 19:45:10.025083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.019946    5958 generic.go:356] "Generic (PLEG): container finished" podID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" containerID="51f02fed75861dbb07545fe29711f16665f0ad2b09b718fce14a38b01e9a5ebd" exitCode=0
Oct 31 19:45:10.025083 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.020283    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" event={"ID":"74d29103-0cb9-4b84-8a7d-ff59d0da9e43","Type":"ContainerDied","Data":"51f02fed75861dbb07545fe29711f16665f0ad2b09b718fce14a38b01e9a5ebd"}
Oct 31 19:45:10.093032 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.092999    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:45:10.131687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.131648    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"]
Oct 31 19:45:10.132922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.132895    5958 cpu_manager.go:414] "containerMap: removing container" podUID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" containerName="maas-api"
Oct 31 19:45:10.132922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.132923    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" containerName="maas-api"
Oct 31 19:45:10.133851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.133290    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" containerName="maas-api"
Oct 31 19:45:10.138419 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.138393    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.155879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.155845    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"]
Oct 31 19:45:10.290249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.290146    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs\") pod \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") "
Oct 31 19:45:10.291619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.291592    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\") pod \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") "
Oct 31 19:45:10.291929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.291909    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca\") pod \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") "
Oct 31 19:45:10.292151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.292130    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc\" (UniqueName: \"kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc\") pod \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\" (UID: \"74d29103-0cb9-4b84-8a7d-ff59d0da9e43\") "
Oct 31 19:45:10.306579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.293134    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.306890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.301502    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca" pod "74d29103-0cb9-4b84-8a7d-ff59d0da9e43" (UID: "74d29103-0cb9-4b84-8a7d-ff59d0da9e43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:10.307202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.307176    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.309189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.308524    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpx4n\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.311685 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.311647    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.312141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.311948    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.312403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.312375    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.312725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.312707    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.312915 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.312900    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.313056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.313042    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.313414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.313398    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:10.317811 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.317774    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc" pod "74d29103-0cb9-4b84-8a7d-ff59d0da9e43" (UID: "74d29103-0cb9-4b84-8a7d-ff59d0da9e43"). InnerVolumeSpecName "kube-api-access-tgjgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:10.318111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.318046    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs" pod "74d29103-0cb9-4b84-8a7d-ff59d0da9e43" (UID: "74d29103-0cb9-4b84-8a7d-ff59d0da9e43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:10.326741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.326698    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls" pod "74d29103-0cb9-4b84-8a7d-ff59d0da9e43" (UID: "74d29103-0cb9-4b84-8a7d-ff59d0da9e43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:10.415391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.415353    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.415590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.415497    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpx4n\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.415694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.415675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.415865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.415844    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.415965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.415949    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416047 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.415848    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416147 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416130    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416195    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416219    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416311 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416289    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416395    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:10.416463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416411    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgjgc\" (UniqueName: \"kubernetes.io/projected/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-kube-api-access-tgjgc\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:10.416463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416424    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74d29103-0cb9-4b84-8a7d-ff59d0da9e43-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:10.416591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416461    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416566    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416703    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.416962 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.416940    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.418049 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.418030    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.418345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.418328    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.429008 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.428975    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.429149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.429128    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpx4n\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.469555 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.469512    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:10.733369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.733330    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"]
Oct 31 19:45:10.749790 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.746163    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.753020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.752980    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"]
Oct 31 19:45:10.753416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.753393    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8dcfa8\"/\"test-model-8dcfa8-kserve-self-signed-certs\""
Oct 31 19:45:10.756643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.756619    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8dcfa8\"/\"openshift-service-ca.crt\""
Oct 31 19:45:10.756785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.756616    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8dcfa8\"/\"kube-root-ca.crt\""
Oct 31 19:45:10.756785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.756649    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8dcfa8\"/\"default-dockercfg-hbfwg\""
Oct 31 19:45:10.777992 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:10.777940    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda619cde9_79d6_40f4_a796_91f16966cc8a.slice/crio-bf2323749b8ee4b248e84eb64f4a2a1b451efa4539a1fbbc9158922ecdda99af WatchSource:0}: Error finding container bf2323749b8ee4b248e84eb64f4a2a1b451efa4539a1fbbc9158922ecdda99af: Status 404 returned error can't find the container with id bf2323749b8ee4b248e84eb64f4a2a1b451efa4539a1fbbc9158922ecdda99af
Oct 31 19:45:10.782435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.782396    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:10.782644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.782484    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:10.782644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.782528    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:10.785966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.785939    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"]
Oct 31 19:45:10.826988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.826950    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.826988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.826992    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.827230 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.827202    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.827524 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.827500    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.827767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.827745    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hcxj\" (UniqueName: \"kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.827908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.827883    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.963372    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.973043    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hcxj\" (UniqueName: \"kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.973156    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.973208    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.973230    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.973372    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.979570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.973962    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.985237 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.985204    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.985509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.985378    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:10.991242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.990783    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:11.008657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:10.998943    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:11.041682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.034098    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hcxj\" (UniqueName: \"kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj\") pod \"test-model-8dcfa8-kserve-5dd474c887-rstpn\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") " pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:11.071379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.070911    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:11.128797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.127348    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" event={"ID":"a619cde9-79d6-40f4-a796-91f16966cc8a","Type":"ContainerStarted","Data":"fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1"}
Oct 31 19:45:11.128797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.127417    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" event={"ID":"a619cde9-79d6-40f4-a796-91f16966cc8a","Type":"ContainerStarted","Data":"bf2323749b8ee4b248e84eb64f4a2a1b451efa4539a1fbbc9158922ecdda99af"}
Oct 31 19:45:11.221919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.201095    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" event={"ID":"74d29103-0cb9-4b84-8a7d-ff59d0da9e43","Type":"ContainerDied","Data":"d591503fca9c3ccfb9c7b0bcfaaafe42cc334a3e3f652d653956585ba242a8a7"}
Oct 31 19:45:11.221919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.201172    5958 scope.go:122] "RemoveContainer" containerID="51f02fed75861dbb07545fe29711f16665f0ad2b09b718fce14a38b01e9a5ebd"
Oct 31 19:45:11.221919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.201393    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"
Oct 31 19:45:11.269866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.269782    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podStartSLOduration=9.269762049 podStartE2EDuration="9.269762049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:11.18006905 +0000 UTC m=+1504.641443666" watchObservedRunningTime="2026-10-31 19:45:11.269762049 +0000 UTC m=+1504.731136666"
Oct 31 19:45:11.270391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.270367    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"]
Oct 31 19:45:11.301813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.301779    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r"]
Oct 31 19:45:11.476600 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.474846    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:11.480951 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.480914    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:11.481102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.480979    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:11.666443 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:11.666361    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5394a82b_db7b_4626_957b_ea3b1b015613.slice/crio-8ff776690fc3bfa694d3bd9c01f89286796b20509c862c8eae9edcbe892a8593 WatchSource:0}: Error finding container 8ff776690fc3bfa694d3bd9c01f89286796b20509c862c8eae9edcbe892a8593: Status 404 returned error can't find the container with id 8ff776690fc3bfa694d3bd9c01f89286796b20509c862c8eae9edcbe892a8593
Oct 31 19:45:11.668420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:11.668383    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"]
Oct 31 19:45:12.241337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:12.241294    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn" event={"ID":"5394a82b-db7b-4626-957b-ea3b1b015613","Type":"ContainerStarted","Data":"a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9"}
Oct 31 19:45:12.241337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:12.241351    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn" event={"ID":"5394a82b-db7b-4626-957b-ea3b1b015613","Type":"ContainerStarted","Data":"8ff776690fc3bfa694d3bd9c01f89286796b20509c862c8eae9edcbe892a8593"}
Oct 31 19:45:12.475698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:12.471005    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:12.475698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:12.471069    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:12.671840 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:12.671803    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" path="/var/lib/kubelet/pods/74d29103-0cb9-4b84-8a7d-ff59d0da9e43/volumes"
Oct 31 19:45:12.897718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:12.897667    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a813e2-7dbcb6dd4d-zmf6r" podUID="74d29103-0cb9-4b84-8a7d-ff59d0da9e43" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.162:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:13.071931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:13.071837    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:13.351621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:13.345908    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:45:13.406567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:13.402042    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn" podStartSLOduration=3.402017814 podStartE2EDuration="3.402017814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:12.29950965 +0000 UTC m=+1505.760884267" watchObservedRunningTime="2026-10-31 19:45:13.402017814 +0000 UTC m=+1506.863392431"
Oct 31 19:45:13.473531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:13.473036    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:13.473531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:13.473113    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:14.521571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:14.510713    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:14.521571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:14.510785    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:15.477566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:15.477384    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:15.477566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:15.477450    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:16.470345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:16.470311    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:16.470788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:16.470364    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:16.662873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:16.662844    5958 scope.go:122] "RemoveContainer" containerID="690902cb04bff94a860ad1313160c87042759a44dde2cc59efa123bacbd6bf70"
Oct 31 19:45:16.663067 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:16.663046    5958 scope.go:122] "RemoveContainer" containerID="22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332"
Oct 31 19:45:16.663219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:16.663198    5958 scope.go:122] "RemoveContainer" containerID="531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f"
Oct 31 19:45:17.063917 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:17.063877    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:45:17.470823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:17.470791    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 31 19:45:17.471220 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:17.470850    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 31 19:45:17.658110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:17.658077    5958 scope.go:122] "RemoveContainer" containerID="33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17"
Oct 31 19:45:18.397231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.397200    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerID="e5f934cbaf28d21140f051d31834e2b6efe643329f74ef518bd68058ff963668" exitCode=60
Oct 31 19:45:18.397377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.397273    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerDied","Data":"e5f934cbaf28d21140f051d31834e2b6efe643329f74ef518bd68058ff963668"}
Oct 31 19:45:18.397377 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.397344    5958 scope.go:122] "RemoveContainer" containerID="22cb2d29f499d8e8fc092e0fc53e368513b8517373d1c4ad02fb7e47cb912332"
Oct 31 19:45:18.397953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.397926    5958 scope.go:122] "RemoveContainer" containerID="e5f934cbaf28d21140f051d31834e2b6efe643329f74ef518bd68058ff963668"
Oct 31 19:45:18.398517 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:18.398443    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw_odh-ai-gateway-infra(fc564e69-d622-4aec-b65a-5ae8c86682b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8"
Oct 31 19:45:18.399609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.399583    5958 generic.go:356] "Generic (PLEG): container finished" podID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerID="daec08701158658a4cfcc6036139f4adafe5263258db9984bf6df1e54a6a8b9f" exitCode=60
Oct 31 19:45:18.399706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.399672    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" event={"ID":"02074908-0ac2-49f0-a639-69dc35a0b5f6","Type":"ContainerDied","Data":"daec08701158658a4cfcc6036139f4adafe5263258db9984bf6df1e54a6a8b9f"}
Oct 31 19:45:18.400129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.400114    5958 scope.go:122] "RemoveContainer" containerID="daec08701158658a4cfcc6036139f4adafe5263258db9984bf6df1e54a6a8b9f"
Oct 31 19:45:18.400616 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:18.400535    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm_odh-ai-gateway-infra(02074908-0ac2-49f0-a639-69dc35a0b5f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6"
Oct 31 19:45:18.401939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.401871    5958 generic.go:356] "Generic (PLEG): container finished" podID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerID="a897ab3fb4c0b495c162e46ce9ef2c0d573d0e70210dda6767175969f924d147" exitCode=60
Oct 31 19:45:18.402031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.401936    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" event={"ID":"05a09253-00e3-4a3f-8720-d8ad07fc2d85","Type":"ContainerDied","Data":"a897ab3fb4c0b495c162e46ce9ef2c0d573d0e70210dda6767175969f924d147"}
Oct 31 19:45:18.402434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.402401    5958 scope.go:122] "RemoveContainer" containerID="a897ab3fb4c0b495c162e46ce9ef2c0d573d0e70210dda6767175969f924d147"
Oct 31 19:45:18.402814 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:18.402793    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8_odh-ai-gateway-infra(05a09253-00e3-4a3f-8720-d8ad07fc2d85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85"
Oct 31 19:45:18.404441 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.404416    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerStarted","Data":"706613618b899a2d6872bdff9347c4d970128201fc887d78c40be8f94f3ac91f"}
Oct 31 19:45:18.414454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.414433    5958 scope.go:122] "RemoveContainer" containerID="690902cb04bff94a860ad1313160c87042759a44dde2cc59efa123bacbd6bf70"
Oct 31 19:45:18.435397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.434966    5958 scope.go:122] "RemoveContainer" containerID="531e25b3c66b3ed3847878539f67e6d8cc78f03407b37851545aebcd7688e84f"
Oct 31 19:45:18.476235 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.476206    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:18.476660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.476513    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:18.477341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.477320    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:45:18.665887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.656825    5958 scope.go:122] "RemoveContainer" containerID="79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1"
Oct 31 19:45:18.665887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.658184    5958 scope.go:122] "RemoveContainer" containerID="235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4"
Oct 31 19:45:18.665887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.659189    5958 scope.go:122] "RemoveContainer" containerID="92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd"
Oct 31 19:45:18.665887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.660624    5958 scope.go:122] "RemoveContainer" containerID="12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd"
Oct 31 19:45:18.665887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:18.662895    5958 scope.go:122] "RemoveContainer" containerID="4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648"
Oct 31 19:45:19.432157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.432114    5958 generic.go:356] "Generic (PLEG): container finished" podID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerID="706613618b899a2d6872bdff9347c4d970128201fc887d78c40be8f94f3ac91f" exitCode=60
Oct 31 19:45:19.432470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.432437    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerDied","Data":"706613618b899a2d6872bdff9347c4d970128201fc887d78c40be8f94f3ac91f"}
Oct 31 19:45:19.432635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.432618    5958 scope.go:122] "RemoveContainer" containerID="33546f13195db06e14a6514697f7bdb950047ec2b34a07b03b1ce6676ece7e17"
Oct 31 19:45:19.433122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.433103    5958 scope.go:122] "RemoveContainer" containerID="706613618b899a2d6872bdff9347c4d970128201fc887d78c40be8f94f3ac91f"
Oct 31 19:45:19.433529 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:19.433505    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch_odh-ai-gateway-infra(367ea9de-5a5c-4006-9bef-80e040470bf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2"
Oct 31 19:45:19.474882 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.474851    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"]
Oct 31 19:45:19.523274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.523180    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"]
Oct 31 19:45:19.580446 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.580409    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"]
Oct 31 19:45:19.598031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.598002    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"]
Oct 31 19:45:19.730558 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.730518    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:19.776826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.776786    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"]
Oct 31 19:45:19.778950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.778908    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:19.779128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.779111    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:19.779271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.779258    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:19.779391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.779376    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:19.779764 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.779733    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:19.779952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.779928    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:19.800672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.800642    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"]
Oct 31 19:45:19.801042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.800971    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.809853 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.809064    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9efeae-openshift-default-dockercfg-jgppr\""
Oct 31 19:45:19.837655 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.836144    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca\") pod \"fc564e69-d622-4aec-b65a-5ae8c86682b8\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") "
Oct 31 19:45:19.837655 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.836255    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f\" (UniqueName: \"kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f\") pod \"fc564e69-d622-4aec-b65a-5ae8c86682b8\" (UID: \"fc564e69-d622-4aec-b65a-5ae8c86682b8\") "
Oct 31 19:45:19.838440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.838409    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca" pod "fc564e69-d622-4aec-b65a-5ae8c86682b8" (UID: "fc564e69-d622-4aec-b65a-5ae8c86682b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:19.860721 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.852930    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f" pod "fc564e69-d622-4aec-b65a-5ae8c86682b8" (UID: "fc564e69-d622-4aec-b65a-5ae8c86682b8"). InnerVolumeSpecName "kube-api-access-48k4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.940695    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.940770    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.940848    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.940878    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.940982    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.941037    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.941074    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.941293    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmrk\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.941402    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.941500    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc564e69-d622-4aec-b65a-5ae8c86682b8-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:19.945305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.941518    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48k4f\" (UniqueName: \"kubernetes.io/projected/fc564e69-d622-4aec-b65a-5ae8c86682b8-kube-api-access-48k4f\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:19.961426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:19.961335    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:20.043209 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.043171    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h\" (UniqueName: \"kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h\") pod \"02074908-0ac2-49f0-a639-69dc35a0b5f6\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") "
Oct 31 19:45:20.043852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.043828    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca\") pod \"02074908-0ac2-49f0-a639-69dc35a0b5f6\" (UID: \"02074908-0ac2-49f0-a639-69dc35a0b5f6\") "
Oct 31 19:45:20.044272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.044243    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca" pod "02074908-0ac2-49f0-a639-69dc35a0b5f6" (UID: "02074908-0ac2-49f0-a639-69dc35a0b5f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:20.044481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.044461    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.044686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.044671    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.044788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.044775    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.044965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.044951    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.045353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.045301    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.045468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.045079    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.045650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.045633    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.045968 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.045950    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmrk\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.046162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046147    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.046388 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046020    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.046490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046472    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.046786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046768    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02074908-0ac2-49f0-a639-69dc35a0b5f6-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:20.046905 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046764    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h" pod "02074908-0ac2-49f0-a639-69dc35a0b5f6" (UID: "02074908-0ac2-49f0-a639-69dc35a0b5f6"). InnerVolumeSpecName "kube-api-access-8272h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:20.047018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046873    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.047157 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046888    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.048566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.046906    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.056306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.056242    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.056462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.056388    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.064276 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.064239    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmrk\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.069338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.069305    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.135630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.135523    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:20.151922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.151885    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8272h\" (UniqueName: \"kubernetes.io/projected/02074908-0ac2-49f0-a639-69dc35a0b5f6-kube-api-access-8272h\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:20.448103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.447692    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:20.535408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.532525    5958 generic.go:356] "Generic (PLEG): container finished" podID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerID="85c3f0b78739cc6e308e2270cb9e37ad255d2dc7eff2a86968d2cabc6aab0944" exitCode=60
Oct 31 19:45:20.535408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.532681    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerDied","Data":"85c3f0b78739cc6e308e2270cb9e37ad255d2dc7eff2a86968d2cabc6aab0944"}
Oct 31 19:45:20.535408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.532741    5958 scope.go:122] "RemoveContainer" containerID="92f60bf8c2e313d81a2d54c82fd714892eff3841fd5869a553274947691370dd"
Oct 31 19:45:20.535408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.533193    5958 scope.go:122] "RemoveContainer" containerID="85c3f0b78739cc6e308e2270cb9e37ad255d2dc7eff2a86968d2cabc6aab0944"
Oct 31 19:45:20.535408 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:20.533576    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh_odh-ai-gateway-infra(839dd42e-ea6a-47e4-b673-ba22730ce33e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e"
Oct 31 19:45:20.541055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.540760    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8" event={"ID":"05a09253-00e3-4a3f-8720-d8ad07fc2d85","Type":"ContainerDied","Data":"16a19336bf760606e96eae9b9632c10abc207491315be5ab68e843e82638d776"}
Oct 31 19:45:20.541055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.540877    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"
Oct 31 19:45:20.552896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.550527    5958 generic.go:356] "Generic (PLEG): container finished" podID="1464acad-4fff-4e70-918c-5c146d76c775" containerID="9b0335fbe4f6f435387b1fd2fdb0321a202aecfaa2276ea31eba88380788ff09" exitCode=60
Oct 31 19:45:20.552896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.550728    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerDied","Data":"9b0335fbe4f6f435387b1fd2fdb0321a202aecfaa2276ea31eba88380788ff09"}
Oct 31 19:45:20.552896 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.551341    5958 scope.go:122] "RemoveContainer" containerID="9b0335fbe4f6f435387b1fd2fdb0321a202aecfaa2276ea31eba88380788ff09"
Oct 31 19:45:20.552896 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:20.552000    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x_odh-ai-gateway-infra(1464acad-4fff-4e70-918c-5c146d76c775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" podUID="1464acad-4fff-4e70-918c-5c146d76c775"
Oct 31 19:45:20.554823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.554793    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw" event={"ID":"fc564e69-d622-4aec-b65a-5ae8c86682b8","Type":"ContainerDied","Data":"55b075d09205b77a73bc79fdc79dce495c6b07cc9d4cd1e5374d641c7aa5944e"}
Oct 31 19:45:20.555162 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.555140    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"
Oct 31 19:45:20.696567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.681167    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk\" (UniqueName: \"kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk\") pod \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") "
Oct 31 19:45:20.696567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.681358    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca\") pod \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\" (UID: \"05a09253-00e3-4a3f-8720-d8ad07fc2d85\") "
Oct 31 19:45:20.696567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.681675    5958 generic.go:356] "Generic (PLEG): container finished" podID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerID="d916e04a017dd0cc489348f8d5e7137f9a11e5d7aa9fded9cf700019dd5f1c0f" exitCode=60
Oct 31 19:45:20.696567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.687082    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca" pod "05a09253-00e3-4a3f-8720-d8ad07fc2d85" (UID: "05a09253-00e3-4a3f-8720-d8ad07fc2d85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:20.794258 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.793840    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a09253-00e3-4a3f-8720-d8ad07fc2d85-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:20.799657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.799603    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk" pod "05a09253-00e3-4a3f-8720-d8ad07fc2d85" (UID: "05a09253-00e3-4a3f-8720-d8ad07fc2d85"). InnerVolumeSpecName "kube-api-access-g9szk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:20.826566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.825827    5958 generic.go:356] "Generic (PLEG): container finished" podID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerID="618a61e43d4f1790d1bc7bb89528e9bcd7147f1c613eaa17034f9741dd91d74e" exitCode=60
Oct 31 19:45:20.855835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.854807    5958 generic.go:356] "Generic (PLEG): container finished" podID="80029600-68bc-428c-adda-705e0b073238" containerID="e29a00a3ad8839381c9454cc7065a4eaeae42c3c9c2035be58489be5b1c31a86" exitCode=60
Oct 31 19:45:20.891825 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:20.887678    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2b36c8_ea98_4c3f_80b7_75cd130861b4.slice/crio-e8a0c19b3e21c1441e1a5df64a92c8e61664c9f6646c25290acc367eb9e65865 WatchSource:0}: Error finding container e8a0c19b3e21c1441e1a5df64a92c8e61664c9f6646c25290acc367eb9e65865: Status 404 returned error can't find the container with id e8a0c19b3e21c1441e1a5df64a92c8e61664c9f6646c25290acc367eb9e65865
Oct 31 19:45:20.910088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.892391    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"
Oct 31 19:45:20.910088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.897706    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9szk\" (UniqueName: \"kubernetes.io/projected/05a09253-00e3-4a3f-8720-d8ad07fc2d85-kube-api-access-g9szk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:20.914964 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.913058    5958 scope.go:122] "RemoveContainer" containerID="a897ab3fb4c0b495c162e46ce9ef2c0d573d0e70210dda6767175969f924d147"
Oct 31 19:45:20.921750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.920476    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:20.921750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.920591    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:20.921750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.920643    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936089    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerDied","Data":"d916e04a017dd0cc489348f8d5e7137f9a11e5d7aa9fded9cf700019dd5f1c0f"}
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936152    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerDied","Data":"618a61e43d4f1790d1bc7bb89528e9bcd7147f1c613eaa17034f9741dd91d74e"}
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936181    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerDied","Data":"e29a00a3ad8839381c9454cc7065a4eaeae42c3c9c2035be58489be5b1c31a86"}
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936216    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"]
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936248    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm" event={"ID":"02074908-0ac2-49f0-a639-69dc35a0b5f6","Type":"ContainerDied","Data":"4b218ae40923c528c530e84e483ada8ec6e657a33cc6864d0771271cfd1ac077"}
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936276    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"]
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.936811    5958 scope.go:122] "RemoveContainer" containerID="e29a00a3ad8839381c9454cc7065a4eaeae42c3c9c2035be58489be5b1c31a86"
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:20.937174    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb_odh-ai-gateway-infra(80029600-68bc-428c-adda-705e0b073238)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" podUID="80029600-68bc-428c-adda-705e0b073238"
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.937672    5958 scope.go:122] "RemoveContainer" containerID="d916e04a017dd0cc489348f8d5e7137f9a11e5d7aa9fded9cf700019dd5f1c0f"
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:20.937976    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg_odh-ai-gateway-infra(432b825c-daf9-4e62-aa2e-9010e92b761f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f"
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.938352    5958 scope.go:122] "RemoveContainer" containerID="618a61e43d4f1790d1bc7bb89528e9bcd7147f1c613eaa17034f9741dd91d74e"
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:20.938655    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn_odh-ai-gateway-infra(65f3803a-fe05-4b98-9c24-c15cb6291361)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361"
Oct 31 19:45:20.941963 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.939967    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891265-dw2vw"]
Oct 31 19:45:20.946753 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.945193    5958 scope.go:122] "RemoveContainer" containerID="12c256d62bc133eae1c5d85b803f81294d451743e4abefe6d47583845a2c2cdd"
Oct 31 19:45:20.957587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.957519    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"]
Oct 31 19:45:20.965618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:20.965469    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-278c04-29891265-kpfj8"]
Oct 31 19:45:21.025247 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.025214    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"]
Oct 31 19:45:21.034917 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.033819    5958 scope.go:122] "RemoveContainer" containerID="e5f934cbaf28d21140f051d31834e2b6efe643329f74ef518bd68058ff963668"
Oct 31 19:45:21.053269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.053154    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9efeae-29891265-hrgqm"]
Oct 31 19:45:21.073756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.073717    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:21.101113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.101083    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:21.105141 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.105116    5958 scope.go:122] "RemoveContainer" containerID="79e40925c0a4a36316f9a23ee703de97d4e99e8746f6b7e58a2f9b968eafa5d1"
Oct 31 19:45:21.146727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.146526    5958 scope.go:122] "RemoveContainer" containerID="235b83c9c37caaf13bbfaa75086cb4acfa417d8b3fd2f4bb999bcb4930631de4"
Oct 31 19:45:21.196791 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.196761    5958 scope.go:122] "RemoveContainer" containerID="4910f80efce0248c5366a7aabe02c0d5e5e9eb299234619f5423f4b13a32c648"
Oct 31 19:45:21.199644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.198285    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:21.215211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.215184    5958 scope.go:122] "RemoveContainer" containerID="daec08701158658a4cfcc6036139f4adafe5263258db9984bf6df1e54a6a8b9f"
Oct 31 19:45:21.308496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.308404    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca\") pod \"367ea9de-5a5c-4006-9bef-80e040470bf2\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") "
Oct 31 19:45:21.308718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.308677    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj\" (UniqueName: \"kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj\") pod \"367ea9de-5a5c-4006-9bef-80e040470bf2\" (UID: \"367ea9de-5a5c-4006-9bef-80e040470bf2\") "
Oct 31 19:45:21.310005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.309971    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca" pod "367ea9de-5a5c-4006-9bef-80e040470bf2" (UID: "367ea9de-5a5c-4006-9bef-80e040470bf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:21.313595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.312815    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj" pod "367ea9de-5a5c-4006-9bef-80e040470bf2" (UID: "367ea9de-5a5c-4006-9bef-80e040470bf2"). InnerVolumeSpecName "kube-api-access-dhdxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:21.412444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.412402    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhdxj\" (UniqueName: \"kubernetes.io/projected/367ea9de-5a5c-4006-9bef-80e040470bf2-kube-api-access-dhdxj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.412444 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.412444    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367ea9de-5a5c-4006-9bef-80e040470bf2-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.572854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.572771    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"]
Oct 31 19:45:21.734585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:21.728098    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"]
Oct 31 19:45:22.141111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.141083    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"]
Oct 31 19:45:22.150748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.150721    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:45:22.158385 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.158356    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"]
Oct 31 19:45:22.167243 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.167207    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" event={"ID":"cd2b36c8-ea98-4c3f-80b7-75cd130861b4","Type":"ContainerStarted","Data":"2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96"}
Oct 31 19:45:22.167415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.167268    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" event={"ID":"cd2b36c8-ea98-4c3f-80b7-75cd130861b4","Type":"ContainerStarted","Data":"e8a0c19b3e21c1441e1a5df64a92c8e61664c9f6646c25290acc367eb9e65865"}
Oct 31 19:45:22.172046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.170468    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"
Oct 31 19:45:22.172046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.171386    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch" event={"ID":"367ea9de-5a5c-4006-9bef-80e040470bf2","Type":"ContainerDied","Data":"9ef0fc00c515c243b555882b2df698d3df8102c300a66c25e526890666c53219"}
Oct 31 19:45:22.172046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.171465    5958 scope.go:122] "RemoveContainer" containerID="706613618b899a2d6872bdff9347c4d970128201fc887d78c40be8f94f3ac91f"
Oct 31 19:45:22.176019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.175996    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"]
Oct 31 19:45:22.369705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.368668    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podStartSLOduration=10.368646385 podStartE2EDuration="10.368646385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:22.36748538 +0000 UTC m=+1515.828860014" watchObservedRunningTime="2026-10-31 19:45:22.368646385 +0000 UTC m=+1515.830021000"
Oct 31 19:45:22.454363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.454036    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"]
Oct 31 19:45:22.464965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.463509    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a4e312-29891265-rcnch"]
Oct 31 19:45:22.667725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.667697    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:22.683938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.680875    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" path="/var/lib/kubelet/pods/02074908-0ac2-49f0-a639-69dc35a0b5f6/volumes"
Oct 31 19:45:22.683938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.681492    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" path="/var/lib/kubelet/pods/05a09253-00e3-4a3f-8720-d8ad07fc2d85/volumes"
Oct 31 19:45:22.683938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.682152    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" path="/var/lib/kubelet/pods/367ea9de-5a5c-4006-9bef-80e040470bf2/volumes"
Oct 31 19:45:22.683938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.683193    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" path="/var/lib/kubelet/pods/fc564e69-d622-4aec-b65a-5ae8c86682b8/volumes"
Oct 31 19:45:22.882565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.874789    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca\") pod \"432b825c-daf9-4e62-aa2e-9010e92b761f\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") "
Oct 31 19:45:22.882565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.874869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2\" (UniqueName: \"kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2\") pod \"432b825c-daf9-4e62-aa2e-9010e92b761f\" (UID: \"432b825c-daf9-4e62-aa2e-9010e92b761f\") "
Oct 31 19:45:22.883064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.883025    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2" pod "432b825c-daf9-4e62-aa2e-9010e92b761f" (UID: "432b825c-daf9-4e62-aa2e-9010e92b761f"). InnerVolumeSpecName "kube-api-access-rbbl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:22.910576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.902969    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca" pod "432b825c-daf9-4e62-aa2e-9010e92b761f" (UID: "432b825c-daf9-4e62-aa2e-9010e92b761f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:22.988375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.988337    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/432b825c-daf9-4e62-aa2e-9010e92b761f-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:22.988375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.988367    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbl2\" (UniqueName: \"kubernetes.io/projected/432b825c-daf9-4e62-aa2e-9010e92b761f-kube-api-access-rbbl2\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:22.990527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.990508    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:22.999276 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:22.999247    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:23.013950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.013917    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:23.027319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.027282    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:23.089903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.089858    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q\" (UniqueName: \"kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q\") pod \"65f3803a-fe05-4b98-9c24-c15cb6291361\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") "
Oct 31 19:45:23.090130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.089964    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca\") pod \"1464acad-4fff-4e70-918c-5c146d76c775\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") "
Oct 31 19:45:23.090130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.090011    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca\") pod \"839dd42e-ea6a-47e4-b673-ba22730ce33e\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") "
Oct 31 19:45:23.090130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.090056    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7\" (UniqueName: \"kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7\") pod \"839dd42e-ea6a-47e4-b673-ba22730ce33e\" (UID: \"839dd42e-ea6a-47e4-b673-ba22730ce33e\") "
Oct 31 19:45:23.090130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.090100    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca\") pod \"80029600-68bc-428c-adda-705e0b073238\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") "
Oct 31 19:45:23.090410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.090250    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6\" (UniqueName: \"kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6\") pod \"1464acad-4fff-4e70-918c-5c146d76c775\" (UID: \"1464acad-4fff-4e70-918c-5c146d76c775\") "
Oct 31 19:45:23.090468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.090443    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca\") pod \"65f3803a-fe05-4b98-9c24-c15cb6291361\" (UID: \"65f3803a-fe05-4b98-9c24-c15cb6291361\") "
Oct 31 19:45:23.090517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.090469    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq\" (UniqueName: \"kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq\") pod \"80029600-68bc-428c-adda-705e0b073238\" (UID: \"80029600-68bc-428c-adda-705e0b073238\") "
Oct 31 19:45:23.092126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.092085    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca" pod "1464acad-4fff-4e70-918c-5c146d76c775" (UID: "1464acad-4fff-4e70-918c-5c146d76c775"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:23.093819 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.093776    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq" pod "80029600-68bc-428c-adda-705e0b073238" (UID: "80029600-68bc-428c-adda-705e0b073238"). InnerVolumeSpecName "kube-api-access-jdzdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:23.094228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.094193    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca" pod "65f3803a-fe05-4b98-9c24-c15cb6291361" (UID: "65f3803a-fe05-4b98-9c24-c15cb6291361"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:23.096878 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.096842    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca" pod "80029600-68bc-428c-adda-705e0b073238" (UID: "80029600-68bc-428c-adda-705e0b073238"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:23.097061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.097034    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca" pod "839dd42e-ea6a-47e4-b673-ba22730ce33e" (UID: "839dd42e-ea6a-47e4-b673-ba22730ce33e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:23.097389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.097335    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7" pod "839dd42e-ea6a-47e4-b673-ba22730ce33e" (UID: "839dd42e-ea6a-47e4-b673-ba22730ce33e"). InnerVolumeSpecName "kube-api-access-hthv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:23.099150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.099119    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6" pod "1464acad-4fff-4e70-918c-5c146d76c775" (UID: "1464acad-4fff-4e70-918c-5c146d76c775"). InnerVolumeSpecName "kube-api-access-b8wp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:23.099279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.099134    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q" pod "65f3803a-fe05-4b98-9c24-c15cb6291361" (UID: "65f3803a-fe05-4b98-9c24-c15cb6291361"). InnerVolumeSpecName "kube-api-access-l2x5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:23.139805 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.139763    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:23.141526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.141491    5958 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 31 19:45:23.141700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.141567    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 31 19:45:23.186932 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.186905    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"
Oct 31 19:45:23.187126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.186936    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg" event={"ID":"432b825c-daf9-4e62-aa2e-9010e92b761f","Type":"ContainerDied","Data":"9eeebb886d662904f853fff194d68dd3983b9f04be849b2daf705464962502a9"}
Oct 31 19:45:23.187126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.187015    5958 scope.go:122] "RemoveContainer" containerID="d916e04a017dd0cc489348f8d5e7137f9a11e5d7aa9fded9cf700019dd5f1c0f"
Oct 31 19:45:23.188782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.188758    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"
Oct 31 19:45:23.188885 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.188747    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn" event={"ID":"65f3803a-fe05-4b98-9c24-c15cb6291361","Type":"ContainerDied","Data":"2c2fa151fe92921f868ad7d29d56f92ee222d8fb65dc075e26a9efbeab007127"}
Oct 31 19:45:23.190682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.190648    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"
Oct 31 19:45:23.191021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.190729    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb" event={"ID":"80029600-68bc-428c-adda-705e0b073238","Type":"ContainerDied","Data":"67a4d2b4b6b9a914a474883276a06815d5cd4ead3d6b576e84cd1984dce14fb6"}
Oct 31 19:45:23.193818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193793    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wp6\" (UniqueName: \"kubernetes.io/projected/1464acad-4fff-4e70-918c-5c146d76c775-kube-api-access-b8wp6\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193817    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3803a-fe05-4b98-9c24-c15cb6291361-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193828    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdzdq\" (UniqueName: \"kubernetes.io/projected/80029600-68bc-428c-adda-705e0b073238-kube-api-access-jdzdq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193839    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2x5q\" (UniqueName: \"kubernetes.io/projected/65f3803a-fe05-4b98-9c24-c15cb6291361-kube-api-access-l2x5q\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193850    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1464acad-4fff-4e70-918c-5c146d76c775-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193859    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839dd42e-ea6a-47e4-b673-ba22730ce33e-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193867    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hthv7\" (UniqueName: \"kubernetes.io/projected/839dd42e-ea6a-47e4-b673-ba22730ce33e-kube-api-access-hthv7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.193994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.193875    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80029600-68bc-428c-adda-705e0b073238-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:23.195000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.194975    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh" event={"ID":"839dd42e-ea6a-47e4-b673-ba22730ce33e","Type":"ContainerDied","Data":"36c884dcb15717ee1fc8ada517a9936979d954db46ad1d684ccc4844e852550d"}
Oct 31 19:45:23.195601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.195563    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"
Oct 31 19:45:23.199889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.198753    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"
Oct 31 19:45:23.199889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.199045    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x" event={"ID":"1464acad-4fff-4e70-918c-5c146d76c775","Type":"ContainerDied","Data":"56a8f8db8839ffc772b809863b2c042ca7e90994cd61278c3da65a5ce62bfb9c"}
Oct 31 19:45:23.240690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.234982    5958 scope.go:122] "RemoveContainer" containerID="618a61e43d4f1790d1bc7bb89528e9bcd7147f1c613eaa17034f9741dd91d74e"
Oct 31 19:45:23.266428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.266403    5958 scope.go:122] "RemoveContainer" containerID="e29a00a3ad8839381c9454cc7065a4eaeae42c3c9c2035be58489be5b1c31a86"
Oct 31 19:45:23.280405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.280379    5958 scope.go:122] "RemoveContainer" containerID="85c3f0b78739cc6e308e2270cb9e37ad255d2dc7eff2a86968d2cabc6aab0944"
Oct 31 19:45:23.290189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.290161    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"]
Oct 31 19:45:23.296195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.296171    5958 scope.go:122] "RemoveContainer" containerID="9b0335fbe4f6f435387b1fd2fdb0321a202aecfaa2276ea31eba88380788ff09"
Oct 31 19:45:23.297122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.297092    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891265-rk4cb"]
Oct 31 19:45:23.338589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.338510    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"]
Oct 31 19:45:23.343775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.343738    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d45f6c-29891265-kll7x"]
Oct 31 19:45:23.360340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.360312    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"]
Oct 31 19:45:23.365468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.365443    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f8c3f7-29891265-f64qn"]
Oct 31 19:45:23.380908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.380873    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"]
Oct 31 19:45:23.391110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.391076    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891265-prqsg"]
Oct 31 19:45:23.432769 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.432735    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"]
Oct 31 19:45:23.437770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:23.437741    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b1fae7-29891265-5vsjh"]
Oct 31 19:45:24.136632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.136591    5958 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 31 19:45:24.137156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.136648    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 31 19:45:24.671744 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.671707    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1464acad-4fff-4e70-918c-5c146d76c775" path="/var/lib/kubelet/pods/1464acad-4fff-4e70-918c-5c146d76c775/volumes"
Oct 31 19:45:24.672517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.672494    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" path="/var/lib/kubelet/pods/432b825c-daf9-4e62-aa2e-9010e92b761f/volumes"
Oct 31 19:45:24.673193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.673173    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" path="/var/lib/kubelet/pods/65f3803a-fe05-4b98-9c24-c15cb6291361/volumes"
Oct 31 19:45:24.674504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.674478    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80029600-68bc-428c-adda-705e0b073238" path="/var/lib/kubelet/pods/80029600-68bc-428c-adda-705e0b073238/volumes"
Oct 31 19:45:24.675158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:24.675136    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" path="/var/lib/kubelet/pods/839dd42e-ea6a-47e4-b673-ba22730ce33e/volumes"
Oct 31 19:45:25.136851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:25.136805    5958 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 31 19:45:25.137275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:25.136862    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 31 19:45:26.140566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:26.139921    5958 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 31 19:45:26.140566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:26.139985    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 31 19:45:27.136585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:27.136468    5958 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 31 19:45:27.136585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:27.136525    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 31 19:45:28.139912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:28.138326    5958 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 31 19:45:28.139912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:28.138414    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 31 19:45:29.142697 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:29.142662    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:29.143296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:29.143272    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:29.144031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:29.144000    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:45:35.406566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:35.393613    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.480571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:35.479653    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.480571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:35.480133    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:40.851189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.851153    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"]
Oct 31 19:45:40.852578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852538    5958 cpu_manager.go:414] "containerMap: removing container" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.852725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852585    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.852725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852627    5958 cpu_manager.go:414] "containerMap: removing container" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.852725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852643    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.852725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852667    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.852725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852682    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.852725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852718    5958 cpu_manager.go:414] "containerMap: removing container" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852730    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852747    5958 cpu_manager.go:414] "containerMap: removing container" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852761    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852786    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852799    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852813    5958 cpu_manager.go:414] "containerMap: removing container" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852825    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852835    5958 cpu_manager.go:414] "containerMap: removing container" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852841    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852864    5958 cpu_manager.go:414] "containerMap: removing container" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852871    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852881    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852887    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852903    5958 cpu_manager.go:414] "containerMap: removing container" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852910    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852932    5958 cpu_manager.go:414] "containerMap: removing container" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852939    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852948    5958 cpu_manager.go:414] "containerMap: removing container" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852954    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852971    5958 cpu_manager.go:414] "containerMap: removing container" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852979    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852988    5958 cpu_manager.go:414] "containerMap: removing container" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.852994    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853012    5958 cpu_manager.go:414] "containerMap: removing container" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853021    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853030    5958 cpu_manager.go:414] "containerMap: removing container" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853037    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853047    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853053    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:40.853055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853075    5958 cpu_manager.go:414] "containerMap: removing container" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853082    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853094    5958 cpu_manager.go:414] "containerMap: removing container" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853101    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853118    5958 cpu_manager.go:414] "containerMap: removing container" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853125    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853148    5958 cpu_manager.go:414] "containerMap: removing container" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853154    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853371    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853381    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853392    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853401    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853411    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853419    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853430    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853439    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853451    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853460    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853471    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853481    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853491    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853500    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="80029600-68bc-428c-adda-705e0b073238" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853510    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853518    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc564e69-d622-4aec-b65a-5ae8c86682b8" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853527    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853539    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853564    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f3803a-fe05-4b98-9c24-c15cb6291361" containerName="cleanup"
Oct 31 19:45:40.854638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.853575    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.860630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.860601    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:40.869223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.869192    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-754e0a-dockercfg-zlcpc\""
Oct 31 19:45:40.869997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.869973    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-754e0a\""
Oct 31 19:45:40.870595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.870520    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"]
Oct 31 19:45:40.907976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.907942    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"]
Oct 31 19:45:40.909039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909014    5958 cpu_manager.go:414] "containerMap: removing container" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.909039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909034    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.909242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909080    5958 cpu_manager.go:414] "containerMap: removing container" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.909242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909085    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02074908-0ac2-49f0-a639-69dc35a0b5f6" containerName="cleanup"
Oct 31 19:45:40.909242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909095    5958 cpu_manager.go:414] "containerMap: removing container" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.909242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909100    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.909503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909254    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="367ea9de-5a5c-4006-9bef-80e040470bf2" containerName="cleanup"
Oct 31 19:45:40.909503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909263    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="839dd42e-ea6a-47e4-b673-ba22730ce33e" containerName="cleanup"
Oct 31 19:45:40.909503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909270    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1464acad-4fff-4e70-918c-5c146d76c775" containerName="cleanup"
Oct 31 19:45:40.909503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909283    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a09253-00e3-4a3f-8720-d8ad07fc2d85" containerName="cleanup"
Oct 31 19:45:40.909503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.909292    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="432b825c-daf9-4e62-aa2e-9010e92b761f" containerName="cleanup"
Oct 31 19:45:40.913588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.913570    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:40.926647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:40.926607    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"]
Oct 31 19:45:41.129623 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.026900    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5vq\" (UniqueName: \"kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq\") pod \"payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") " pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.192567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.132743    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.192567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.132851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") " pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.192567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.132920    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnvr\" (UniqueName: \"kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr\") pod \"payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.257006 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.251598    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5vq\" (UniqueName: \"kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq\") pod \"payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") " pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.257006 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.251747    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.394628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.393867    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.414823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.411712    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") " pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.414823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.411804    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnvr\" (UniqueName: \"kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr\") pod \"payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.414823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.413377    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") " pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.425588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.423527    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5vq\" (UniqueName: \"kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq\") pod \"payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") " pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.500836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.492208    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnvr\" (UniqueName: \"kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr\") pod \"payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.630231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.630190    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:41.785429 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.785390    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:41.935955 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:41.935925    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0381a0e_ac17_456e_81fc_d283cb5e62ae.slice/crio-1e49c52d7e815714f70a507162c7cfad828bdad00173a68737839f9355daea71 WatchSource:0}: Error finding container 1e49c52d7e815714f70a507162c7cfad828bdad00173a68737839f9355daea71: Status 404 returned error can't find the container with id 1e49c52d7e815714f70a507162c7cfad828bdad00173a68737839f9355daea71
Oct 31 19:45:41.936484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:41.936457    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"]
Oct 31 19:45:42.084792 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:42.084751    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87973e9e_4ad8_48d1_8714_975d9bad5403.slice/crio-38002fcee8c6d2ea8258971fef2816645abbfe2c601dc3cd5bce2cbf92fdbca8 WatchSource:0}: Error finding container 38002fcee8c6d2ea8258971fef2816645abbfe2c601dc3cd5bce2cbf92fdbca8: Status 404 returned error can't find the container with id 38002fcee8c6d2ea8258971fef2816645abbfe2c601dc3cd5bce2cbf92fdbca8
Oct 31 19:45:42.086564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:42.086526    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"]
Oct 31 19:45:43.008642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.008601    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" event={"ID":"87973e9e-4ad8-48d1-8714-975d9bad5403","Type":"ContainerStarted","Data":"1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c"}
Oct 31 19:45:43.009249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.009223    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" event={"ID":"87973e9e-4ad8-48d1-8714-975d9bad5403","Type":"ContainerStarted","Data":"38002fcee8c6d2ea8258971fef2816645abbfe2c601dc3cd5bce2cbf92fdbca8"}
Oct 31 19:45:43.009738 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.009713    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:43.016000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.014833    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" event={"ID":"f0381a0e-ac17-456e-81fc-d283cb5e62ae","Type":"ContainerStarted","Data":"4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4"}
Oct 31 19:45:43.016000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.014880    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" event={"ID":"f0381a0e-ac17-456e-81fc-d283cb5e62ae","Type":"ContainerStarted","Data":"1e49c52d7e815714f70a507162c7cfad828bdad00173a68737839f9355daea71"}
Oct 31 19:45:43.016000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.014991    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:43.038507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.038457    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" podStartSLOduration=3.038444468 podStartE2EDuration="3.038444468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:43.032029696 +0000 UTC m=+1536.493404338" watchObservedRunningTime="2026-10-31 19:45:43.038444468 +0000 UTC m=+1536.499819083"
Oct 31 19:45:43.065319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:43.065267    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" podStartSLOduration=3.065253393 podStartE2EDuration="3.065253393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:43.060609349 +0000 UTC m=+1536.521983961" watchObservedRunningTime="2026-10-31 19:45:43.065253393 +0000 UTC m=+1536.526628007"
Oct 31 19:45:47.048234 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:47.048150    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:45:50.224649 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:50.224605    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"]
Oct 31 19:45:50.225088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:50.224882    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" containerID="cri-o://c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7" gracePeriod=30
Oct 31 19:45:54.026953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:54.026921    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:45:54.027441 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:54.027236    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:45:55.104466 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.104419    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:45:55.104965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.104511    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:45:55.596930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.596898    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:45:55.688218 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.688128    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.688531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.688506    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.688854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.688834    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.689040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.689026    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.696576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.689309    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.696874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.696849    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.697062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.697045    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.697188 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.697173    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.697365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.697349    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data\") pod \"1ecffc5b-8793-4213-a825-7c78d1333e79\" (UID: \"1ecffc5b-8793-4213-a825-7c78d1333e79\") "
Oct 31 19:45:55.701507 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.701476    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:55.702782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.702752    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:55.703064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.703045    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:55.703355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.703334    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:55.704037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.704013    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:55.706612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.706587    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:55.709016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.708979    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:45:55.713261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.713224    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:55.713536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.713507    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh" pod "1ecffc5b-8793-4213-a825-7c78d1333e79" (UID: "1ecffc5b-8793-4213-a825-7c78d1333e79"). InnerVolumeSpecName "kube-api-access-g5xxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:55.725914 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.725614    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"]
Oct 31 19:45:55.727287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.727262    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy"
Oct 31 19:45:55.727417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.727289    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy"
Oct 31 19:45:55.727707 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.727599    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerName="istio-proxy"
Oct 31 19:45:55.739587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.733938    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"]
Oct 31 19:45:55.739587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.734053    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.770441    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-03dfee-openshift-default-dockercfg-nfkrf\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.809961    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810022    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810114    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwzd\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810170    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810308    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810470    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810587    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810654    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810678    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810797    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810813    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810829    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xxh\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-kube-api-access-g5xxh\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810843    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810857    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810870    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810884    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecffc5b-8793-4213-a825-7c78d1333e79-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810897    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecffc5b-8793-4213-a825-7c78d1333e79-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.811569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.810909    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecffc5b-8793-4213-a825-7c78d1333e79-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.945815    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.945901    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.945927    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.945990    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.946033    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.946119    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwzd\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.946178    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.946323    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.946625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.946498    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.947195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.947134    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.961424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.947388    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.961424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.947773    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.961424 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.948300    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.967622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.963880    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.967797 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.967750    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.980430 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.979446    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.980890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.980859    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:55.983800 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:55.983773    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwzd\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:56.066794 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.066758    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"]
Oct 31 19:45:56.074018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.073993    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.087182 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.087150    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"]
Oct 31 19:45:56.087981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.087953    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-785d26\"/\"default-dockercfg-9hkzq\""
Oct 31 19:45:56.088155 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.088130    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-785d26\"/\"test-model-785d26-kserve-self-signed-certs\""
Oct 31 19:45:56.102862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.102830    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-785d26\"/\"openshift-service-ca.crt\""
Oct 31 19:45:56.103039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.102876    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-785d26\"/\"kube-root-ca.crt\""
Oct 31 19:45:56.154287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.154252    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.154725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.154439    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.154725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.154646    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.154725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.154689    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzm8\" (UniqueName: \"kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.154828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.154777    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.154828 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.154810    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.196457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.196417    5958 generic.go:356] "Generic (PLEG): container finished" podID="1ecffc5b-8793-4213-a825-7c78d1333e79" containerID="c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7" exitCode=0
Oct 31 19:45:56.196730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.196490    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" event={"ID":"1ecffc5b-8793-4213-a825-7c78d1333e79","Type":"ContainerDied","Data":"c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7"}
Oct 31 19:45:56.196730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.196634    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj" event={"ID":"1ecffc5b-8793-4213-a825-7c78d1333e79","Type":"ContainerDied","Data":"3e7b60ac6e303c0c24ccfea55a0da2fe3318337a8b8db5ea7d78607b4ff6cd41"}
Oct 31 19:45:56.196730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.196668    5958 scope.go:122] "RemoveContainer" containerID="c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7"
Oct 31 19:45:56.196927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.196502    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"
Oct 31 19:45:56.219999 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.219977    5958 scope.go:122] "RemoveContainer" containerID="c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7"
Oct 31 19:45:56.220343 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:45:56.220323    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7\": container with ID starting with c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7 not found: ID does not exist" containerID="c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7"
Oct 31 19:45:56.220413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.220356    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7"} err="failed to get container status \"c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7\": rpc error: code = NotFound desc = could not find container \"c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7\": container with ID starting with c2569ffa92d0df3dccef05a0e2477dba7f7828ea7e521acb52bc93893c483ef7 not found: ID does not exist"
Oct 31 19:45:56.236057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.236018    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"]
Oct 31 19:45:56.242437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.242400    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-pwjxj"]
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.270301    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.270393    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzm8\" (UniqueName: \"kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.270450    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.270474    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.270726    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.270857    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.277574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.272161    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.292567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.281636    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.308318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.293789    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:56.309015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.308986    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.318621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.315334    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzm8\" (UniqueName: \"kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.333562 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.331878    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.351570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.343262    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs\") pod \"test-model-785d26-kserve-5b9cfbff79-l7zhq\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") " pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.402696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.396534    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:56.683005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.682273    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecffc5b-8793-4213-a825-7c78d1333e79" path="/var/lib/kubelet/pods/1ecffc5b-8793-4213-a825-7c78d1333e79/volumes"
Oct 31 19:45:56.956763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.956729    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"]
Oct 31 19:45:56.961795 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:45:56.961771    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a739028_c5f1_4fc2_98fb_68800d3269f3.slice/crio-4ea5adcfb9a45d69ccb5e771ff867b07a0b58b62d724f7cb892129482e017298 WatchSource:0}: Error finding container 4ea5adcfb9a45d69ccb5e771ff867b07a0b58b62d724f7cb892129482e017298: Status 404 returned error can't find the container with id 4ea5adcfb9a45d69ccb5e771ff867b07a0b58b62d724f7cb892129482e017298
Oct 31 19:45:56.963934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.963911    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"]
Oct 31 19:45:56.964625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.964599    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:56.964675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.964658    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:56.964715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:56.964687    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:45:57.211458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.211369    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" event={"ID":"1a739028-c5f1-4fc2-98fb-68800d3269f3","Type":"ContainerStarted","Data":"cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6"}
Oct 31 19:45:57.211458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.211434    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" event={"ID":"1a739028-c5f1-4fc2-98fb-68800d3269f3","Type":"ContainerStarted","Data":"4ea5adcfb9a45d69ccb5e771ff867b07a0b58b62d724f7cb892129482e017298"}
Oct 31 19:45:57.214787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.214755    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq" event={"ID":"9f21647a-426d-44b2-a060-e2c0e4a0ce53","Type":"ContainerStarted","Data":"ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1"}
Oct 31 19:45:57.214924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.214801    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq" event={"ID":"9f21647a-426d-44b2-a060-e2c0e4a0ce53","Type":"ContainerStarted","Data":"10b613be3783531deaeceaf3a99a61c625a14dc4e978ee57e8e40b917fa2420d"}
Oct 31 19:45:57.242403 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.242342    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podStartSLOduration=50.242323678 podStartE2EDuration="50.242323678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:57.237745552 +0000 UTC m=+1550.699120190" watchObservedRunningTime="2026-10-31 19:45:57.242323678 +0000 UTC m=+1550.703698294"
Oct 31 19:45:57.264248 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.264187    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq" podStartSLOduration=1.264168349 podStartE2EDuration="1.264168349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:57.260524855 +0000 UTC m=+1550.721899470" watchObservedRunningTime="2026-10-31 19:45:57.264168349 +0000 UTC m=+1550.725542967"
Oct 31 19:45:57.296265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.296229    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:45:57.420497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:57.413720    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:45:58.297701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:58.297662    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:45:58.298119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:58.297731    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:45:59.296659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:59.296623    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:45:59.296862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:45:59.296676    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:46:00.296907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:00.296863    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:46:00.297431 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:00.296934    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:46:01.330565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:01.324698    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:46:01.330565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:01.324763    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:46:02.314570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:02.302364    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:46:02.314570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:02.302430    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:46:03.308586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:03.296898    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:46:03.308586 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:03.296965    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:46:04.296198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:04.296157    5958 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 31 19:46:04.296383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:04.296220    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 31 19:46:05.301654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:05.301619    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:46:05.302113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:05.302094    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:46:05.303024 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:05.302990    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:46:06.404569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:06.402822    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:46:06.464296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:06.464265    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:46:06.476721 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:06.476688    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:46:13.532264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:13.532227    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:46:17.049530 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:46:17.049478    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:46:20.466450 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:20.466413    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"]
Oct 31 19:46:20.466932 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:20.466679    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy" containerID="cri-o://684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9" gracePeriod=30
Oct 31 19:46:20.486416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:20.486381    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"]
Oct 31 19:46:20.486786 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:20.486709    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy" containerID="cri-o://3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66" gracePeriod=30
Oct 31 19:46:25.805860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.805827    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:46:25.859110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.859077    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"]
Oct 31 19:46:25.860365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.860334    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy"
Oct 31 19:46:25.860365 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.860366    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy"
Oct 31 19:46:25.860711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.860685    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerName="istio-proxy"
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.868181    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.874503    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.874732    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.874810    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.874873    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.875102    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.875146    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.875212    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.875257    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.875362    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket\") pod \"1026494f-1bd1-4171-96cb-f4322fe75f65\" (UID: \"1026494f-1bd1-4171-96cb-f4322fe75f65\") "
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.876468    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.876665    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.878368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.876974    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:25.887904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.882536    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b1fae7-openshift-default-dockercfg-nx28t\""
Oct 31 19:46:25.887904 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.887216    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.888256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.888234    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.891852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.891827    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"]
Oct 31 19:46:25.906418 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.906382    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "kube-api-access-jkx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:25.907228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.906320    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.908979 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.908950    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:25.910662 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.909329    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo" pod "1026494f-1bd1-4171-96cb-f4322fe75f65" (UID: "1026494f-1bd1-4171-96cb-f4322fe75f65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:46:25.918534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.918509    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:46:25.959305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.956758    5958 generic.go:356] "Generic (PLEG): container finished" podID="1026494f-1bd1-4171-96cb-f4322fe75f65" containerID="684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9" exitCode=0
Oct 31 19:46:25.959305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.957010    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" event={"ID":"1026494f-1bd1-4171-96cb-f4322fe75f65","Type":"ContainerDied","Data":"684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9"}
Oct 31 19:46:25.959305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.957063    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x" event={"ID":"1026494f-1bd1-4171-96cb-f4322fe75f65","Type":"ContainerDied","Data":"1964d1bf4c27785b8c51b6cafda3e983735779231d3aa31f00e60a678f98a0ca"}
Oct 31 19:46:25.959305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.957092    5958 scope.go:122] "RemoveContainer" containerID="684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9"
Oct 31 19:46:25.959305 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.957276    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"
Oct 31 19:46:25.964943 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.964913    5958 generic.go:356] "Generic (PLEG): container finished" podID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerID="3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66" exitCode=0
Oct 31 19:46:25.965357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.965337    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"
Oct 31 19:46:25.965513 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.965345    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" event={"ID":"b95555db-70a4-4383-bb42-8dae1d5e61e0","Type":"ContainerDied","Data":"3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66"}
Oct 31 19:46:25.965777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.965756    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w" event={"ID":"b95555db-70a4-4383-bb42-8dae1d5e61e0","Type":"ContainerDied","Data":"53f530eadbc687f659390e57f730cfacef465bccb5d20ae55d80bd118c37818b"}
Oct 31 19:46:25.970500 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.970477    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"]
Oct 31 19:46:25.972677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.972654    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy"
Oct 31 19:46:25.972884 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.972808    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy"
Oct 31 19:46:25.973280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.973261    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" containerName="istio-proxy"
Oct 31 19:46:25.981710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.981681    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.981922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.981902    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.982245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.982219    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.982400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.982385    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.982515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.982499    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.982824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.982806    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.983314 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.983291    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.983591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.983571    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.984015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.983997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy\") pod \"b95555db-70a4-4383-bb42-8dae1d5e61e0\" (UID: \"b95555db-70a4-4383-bb42-8dae1d5e61e0\") "
Oct 31 19:46:25.984693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.984671    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.984965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.983723    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:25.984965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.983734    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.984965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.984143    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.984965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.984163    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.984965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.984427    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.985382 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.985363    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.985624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.985605    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.985943 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.985924    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.986442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.986422    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.988123 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.988101    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47dwd\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.988231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.988190    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.988320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.988034    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:25.988448 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.988427    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.990127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.990107    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"]
Oct 31 19:46:25.990386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.990318    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:25.991437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991415    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:25.991760 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991739    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991764    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991780    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkx8j\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-kube-api-access-jkx8j\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991793    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991805    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991818    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1026494f-1bd1-4171-96cb-f4322fe75f65-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991830    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991842    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.991856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991853    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b95555db-70a4-4383-bb42-8dae1d5e61e0-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.992277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991865    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.992277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991877    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.992277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991888    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.992277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991900    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.992277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991913    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1026494f-1bd1-4171-96cb-f4322fe75f65-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:25.992277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:25.991925    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1026494f-1bd1-4171-96cb-f4322fe75f65-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:26.001803 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.001767    5958 scope.go:122] "RemoveContainer" containerID="684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9"
Oct 31 19:46:26.001925 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.001768    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:26.002120 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.002076    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-754e0a-serving-cert\""
Oct 31 19:46:26.004415 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:46:26.004230    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9\": container with ID starting with 684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9 not found: ID does not exist" containerID="684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9"
Oct 31 19:46:26.004415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.004276    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9"} err="failed to get container status \"684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9\": rpc error: code = NotFound desc = could not find container \"684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9\": container with ID starting with 684648026ed8c5ea922e58331d66469fecddcd1b47a88eb33f8cb0ec4c4ccfb9 not found: ID does not exist"
Oct 31 19:46:26.004415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.004301    5958 scope.go:122] "RemoveContainer" containerID="3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66"
Oct 31 19:46:26.004858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.004833    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "kube-api-access-9mjhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:26.006564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.006479    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo" pod "b95555db-70a4-4383-bb42-8dae1d5e61e0" (UID: "b95555db-70a4-4383-bb42-8dae1d5e61e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:46:26.018285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.018228    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"]
Oct 31 19:46:26.026262 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.026216    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-wv98x"]
Oct 31 19:46:26.031454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.030972    5958 scope.go:122] "RemoveContainer" containerID="3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66"
Oct 31 19:46:26.031454 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:46:26.031386    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66\": container with ID starting with 3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66 not found: ID does not exist" containerID="3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66"
Oct 31 19:46:26.031454 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.031425    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66"} err="failed to get container status \"3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66\": rpc error: code = NotFound desc = could not find container \"3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66\": container with ID starting with 3beadb6349064be8b49cfa007dbcbdde6ebf0f25510f29a927d672758ad54b66 not found: ID does not exist"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.162903    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163131    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163257    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163329    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163367    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163524    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllhp\" (UniqueName: \"kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163576    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163616    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163664    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163693    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47dwd\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163727    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163802    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.163864    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.164001    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.164018    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b95555db-70a4-4383-bb42-8dae1d5e61e0-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:26.165575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.164035    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjhd\" (UniqueName: \"kubernetes.io/projected/b95555db-70a4-4383-bb42-8dae1d5e61e0-kube-api-access-9mjhd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:46:26.183323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.174417    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.183323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.175074    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.183323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.175314    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.183323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.178570    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.183323 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.178843    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.188343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.188311    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.188820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.188794    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.191266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.191246    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.191441 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.191411    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47dwd\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.226727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.226694    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:26.265296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.265213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllhp\" (UniqueName: \"kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.265296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.265258    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.265296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.265296    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.265659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.265352    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.266222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.266184    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.268482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.268455    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.268665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.268644    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.279422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.279390    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllhp\" (UniqueName: \"kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp\") pod \"maas-api-e2e-inf-a-754e0a-d5874d588-8294c\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.300168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.300098    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"]
Oct 31 19:46:26.312025 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:46:26.311989    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95555db_70a4_4383_bb42_8dae1d5e61e0.slice/crio-53f530eadbc687f659390e57f730cfacef465bccb5d20ae55d80bd118c37818b\": RecentStats: unable to find data in memory cache]"
Oct 31 19:46:26.314953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.314921    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-r9b7w"]
Oct 31 19:46:26.328318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.328291    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:26.547208 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:46:26.547171    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4d71d6_13c0_439e_a7ba_3405108a2d8c.slice/crio-85b4d09bc776afb74eee9973127bb9b7d95144eaf82df266d6586c8d886e14df WatchSource:0}: Error finding container 85b4d09bc776afb74eee9973127bb9b7d95144eaf82df266d6586c8d886e14df: Status 404 returned error can't find the container with id 85b4d09bc776afb74eee9973127bb9b7d95144eaf82df266d6586c8d886e14df
Oct 31 19:46:26.549808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.549778    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"]
Oct 31 19:46:26.550967 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.550933    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:46:26.551051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.551016    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:46:26.551090 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.551058    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:46:26.638189 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:46:26.638156    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc76981_8b90_47e1_bd2a_c82e9ec017e6.slice/crio-809dc078e519fe03e9a5c1b8cc4d333c84d2e104f2e364720b3853167ed68350 WatchSource:0}: Error finding container 809dc078e519fe03e9a5c1b8cc4d333c84d2e104f2e364720b3853167ed68350: Status 404 returned error can't find the container with id 809dc078e519fe03e9a5c1b8cc4d333c84d2e104f2e364720b3853167ed68350
Oct 31 19:46:26.639990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.639968    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:46:26.640318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.640296    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"]
Oct 31 19:46:26.672899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.672871    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1026494f-1bd1-4171-96cb-f4322fe75f65" path="/var/lib/kubelet/pods/1026494f-1bd1-4171-96cb-f4322fe75f65/volumes"
Oct 31 19:46:26.673479 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:26.673465    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95555db-70a4-4383-bb42-8dae1d5e61e0" path="/var/lib/kubelet/pods/b95555db-70a4-4383-bb42-8dae1d5e61e0/volumes"
Oct 31 19:46:27.011610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.003991    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c" event={"ID":"efc76981-8b90-47e1-bd2a-c82e9ec017e6","Type":"ContainerStarted","Data":"809dc078e519fe03e9a5c1b8cc4d333c84d2e104f2e364720b3853167ed68350"}
Oct 31 19:46:27.011610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.006523    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" event={"ID":"aa4d71d6-13c0-439e-a7ba-3405108a2d8c","Type":"ContainerStarted","Data":"7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d"}
Oct 31 19:46:27.011610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.006600    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" event={"ID":"aa4d71d6-13c0-439e-a7ba-3405108a2d8c","Type":"ContainerStarted","Data":"85b4d09bc776afb74eee9973127bb9b7d95144eaf82df266d6586c8d886e14df"}
Oct 31 19:46:27.077380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.077168    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podStartSLOduration=40.07713567 podStartE2EDuration="40.07713567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:27.052278764 +0000 UTC m=+1580.513653382" watchObservedRunningTime="2026-10-31 19:46:27.07713567 +0000 UTC m=+1580.538510289"
Oct 31 19:46:27.227255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.227218    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:27.229056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.228975    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 19:46:27.229221 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:27.229077    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 19:46:28.039223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:28.039185    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c" event={"ID":"efc76981-8b90-47e1-bd2a-c82e9ec017e6","Type":"ContainerStarted","Data":"fa51681f8ed25fdeac80e506de771f1bb705be0fede7ea0fe586eefcd0494548"}
Oct 31 19:46:28.039899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:28.039875    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:28.077329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:28.077261    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c" podStartSLOduration=23.539717598 podStartE2EDuration="24.077237952s" totalImagesPullingTime="537.520354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:28.064019574 +0000 UTC m=+1581.525394203" watchObservedRunningTime="2026-10-31 19:46:28.077237952 +0000 UTC m=+1581.538612567"
Oct 31 19:46:28.234580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:28.231883    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 19:46:28.234580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:28.231954    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 19:46:29.238886 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:29.228668    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 19:46:29.239580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:29.239517    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 19:46:30.229571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:30.227088    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 19:46:30.229571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:30.227150    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 19:46:31.230020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:31.229969    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 19:46:31.230566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:31.230037    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 19:46:32.231037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:32.230990    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 19:46:32.231527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:32.231059    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 19:46:33.425533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:33.425467    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:33.426089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:33.425986    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:33.427219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:33.427194    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:46:35.082367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:46:35.082334    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:46:47.048333 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:46:47.048296    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:47:17.061835 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:47:17.061754    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:47:47.048166 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:47:47.048122    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:47:49.823929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:47:49.823891    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:48:05.744977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:05.744941    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"]
Oct 31 19:48:05.745482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:05.745223    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy" containerID="cri-o://12c20372e81bd12ab0fa000de0e3bfbffba24991d8086164ce79cc47a8009015" gracePeriod=30
Oct 31 19:48:05.750124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:05.750099    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"]
Oct 31 19:48:05.750314 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:05.750294    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy" containerID="cri-o://b08c15fe3d28932fa541217e0f62919e1d9f3b22368fb7dd08ce167567d5baad" gracePeriod=30
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.222601    5958 generic.go:356] "Generic (PLEG): container finished" podID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerID="12c20372e81bd12ab0fa000de0e3bfbffba24991d8086164ce79cc47a8009015" exitCode=0
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.222714    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" event={"ID":"9787d41b-dc62-49f6-83e8-51ab509f73f1","Type":"ContainerDied","Data":"12c20372e81bd12ab0fa000de0e3bfbffba24991d8086164ce79cc47a8009015"}
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.222765    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9" event={"ID":"9787d41b-dc62-49f6-83e8-51ab509f73f1","Type":"ContainerDied","Data":"f028a0359171b82e251f21289ab82d57c9231f08c5dc2dd981997dc444f74c22"}
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.222788    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f028a0359171b82e251f21289ab82d57c9231f08c5dc2dd981997dc444f74c22"
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.224989    5958 generic.go:356] "Generic (PLEG): container finished" podID="7572f13d-f605-4362-96dc-0c1d5575833d" containerID="b08c15fe3d28932fa541217e0f62919e1d9f3b22368fb7dd08ce167567d5baad" exitCode=0
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.225081    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" event={"ID":"7572f13d-f605-4362-96dc-0c1d5575833d","Type":"ContainerDied","Data":"b08c15fe3d28932fa541217e0f62919e1d9f3b22368fb7dd08ce167567d5baad"}
Oct 31 19:48:11.233788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.231066    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:48:11.313202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.313127    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:48:11.323387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.323354    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"]
Oct 31 19:48:11.331711 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.331684    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy"
Oct 31 19:48:11.331897 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.331882    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy"
Oct 31 19:48:11.332094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.332082    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy"
Oct 31 19:48:11.332598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.332577    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy"
Oct 31 19:48:11.335002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.334978    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" containerName="istio-proxy"
Oct 31 19:48:11.336421 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.336400    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" containerName="istio-proxy"
Oct 31 19:48:11.348750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.348715    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"]
Oct 31 19:48:11.349051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.348906    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.365199 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.365056    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-754e0a-openshift-default-dockercfg-t75h6\""
Oct 31 19:48:11.380233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.380184    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"]
Oct 31 19:48:11.390232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.390201    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"]
Oct 31 19:48:11.390451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.390308    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.414004 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.413951    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.414204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414079    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414144    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414223    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414270    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414342    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.414458 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414417    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414468    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.414582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414499    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414566    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.414774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414597    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414718    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.414774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414762    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.414931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.414834    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.415055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415036    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.415130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415070    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.415265 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415236    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.415664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415633    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data\") pod \"7572f13d-f605-4362-96dc-0c1d5575833d\" (UID: \"7572f13d-f605-4362-96dc-0c1d5575833d\") "
Oct 31 19:48:11.415818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415704    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo\") pod \"9787d41b-dc62-49f6-83e8-51ab509f73f1\" (UID: \"9787d41b-dc62-49f6-83e8-51ab509f73f1\") "
Oct 31 19:48:11.415818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415748    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.415997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.415967    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.416883 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.416450    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.416883 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.416873    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.417055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.416897    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.417138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.417114    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.417207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.417137    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:11.417360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.417339    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.417443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.417387    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.417954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.417925    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:11.418445 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.418418    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.419283 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.419255    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:11.421529 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.421447    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.422675 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.421651    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.422879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.422847    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.423125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.423102    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "kube-api-access-qq5p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.423217 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.423183    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "kube-api-access-lw6gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.423417 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.423400    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo" pod "9787d41b-dc62-49f6-83e8-51ab509f73f1" (UID: "9787d41b-dc62-49f6-83e8-51ab509f73f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:11.423503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.423411    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token" pod "7572f13d-f605-4362-96dc-0c1d5575833d" (UID: "7572f13d-f605-4362-96dc-0c1d5575833d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.518396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518355    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.518723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518441    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.518723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518529    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzrb\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.518723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518669    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.518910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518754    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.518910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518815    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.518910 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518893    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.519062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.518953    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.519118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519068    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.519163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519145    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxqj\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.519212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519174    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.519212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519191    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.519314 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519301    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.519354 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519327    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.519414 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519402    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.519461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519441    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.519498 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519467    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.519595 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519507    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.519724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519708    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7572f13d-f605-4362-96dc-0c1d5575833d-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519730    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6gj\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-kube-api-access-lw6gj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519741    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519749    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519761    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7572f13d-f605-4362-96dc-0c1d5575833d-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519770    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519778    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9787d41b-dc62-49f6-83e8-51ab509f73f1-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519787    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519795    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519804    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519813    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519821    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519829    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9787d41b-dc62-49f6-83e8-51ab509f73f1-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519840    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5p7\" (UniqueName: \"kubernetes.io/projected/9787d41b-dc62-49f6-83e8-51ab509f73f1-kube-api-access-qq5p7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519855    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7572f13d-f605-4362-96dc-0c1d5575833d-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.519930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.519867    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7572f13d-f605-4362-96dc-0c1d5575833d-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.621644 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621522    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.621850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621662    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.621850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621726    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxqj\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.621850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621764    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.621850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621787    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.621850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621838    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621870    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.621976    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622013    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622055    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622129    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622232    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622305    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622336    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzrb\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622367    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622378    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622301    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622457    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622532    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622707    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.622823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622721    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.622823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622371    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.623125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.622907    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.623125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.623117    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.623238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.623122    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.623238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.623117    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.623238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.623165    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.623388 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.623364    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.624698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.624677    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.624874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.624858    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.625207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.625191    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.625322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.625307    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.638638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.638609    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.638784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.638737    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxqj\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.639289 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.639270    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzrb\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.639363 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.639329    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.676423 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.676392    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:11.722039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.709586    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:11.993590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.993536    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"]
Oct 31 19:48:11.994296 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:48:11.994265    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f38528_4ce5_4c6f_8fb4_833bba442ff8.slice/crio-551ebec6bf89abce35738e235a8f402bd0e51961a817831b7b7883729364ac00 WatchSource:0}: Error finding container 551ebec6bf89abce35738e235a8f402bd0e51961a817831b7b7883729364ac00: Status 404 returned error can't find the container with id 551ebec6bf89abce35738e235a8f402bd0e51961a817831b7b7883729364ac00
Oct 31 19:48:11.998096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.998064    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:48:11.998184 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.998131    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:48:11.998184 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:11.998162    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:48:12.126222 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:48:12.125962    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf128bd83_1027_45a5_b750_ce1920e34719.slice/crio-acec092ef69f0de5f9ef913252a01c6e2950c87167cf93ac531c93f2d1aebcc0 WatchSource:0}: Error finding container acec092ef69f0de5f9ef913252a01c6e2950c87167cf93ac531c93f2d1aebcc0: Status 404 returned error can't find the container with id acec092ef69f0de5f9ef913252a01c6e2950c87167cf93ac531c93f2d1aebcc0
Oct 31 19:48:12.126634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.126607    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"]
Oct 31 19:48:12.130223 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.130164    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:48:12.130337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.130247    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:48:12.130337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.130279    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:48:12.238136 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.238105    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" event={"ID":"a5f38528-4ce5-4c6f-8fb4-833bba442ff8","Type":"ContainerStarted","Data":"abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb"}
Oct 31 19:48:12.238687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.238170    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" event={"ID":"a5f38528-4ce5-4c6f-8fb4-833bba442ff8","Type":"ContainerStarted","Data":"551ebec6bf89abce35738e235a8f402bd0e51961a817831b7b7883729364ac00"}
Oct 31 19:48:12.240055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.240035    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"
Oct 31 19:48:12.240185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.240058    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk" event={"ID":"7572f13d-f605-4362-96dc-0c1d5575833d","Type":"ContainerDied","Data":"b1357a660de53803769c9c53e2f50438165f69870a5209904f0565882c46aead"}
Oct 31 19:48:12.240185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.240120    5958 scope.go:122] "RemoveContainer" containerID="b08c15fe3d28932fa541217e0f62919e1d9f3b22368fb7dd08ce167567d5baad"
Oct 31 19:48:12.243372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.242865    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"
Oct 31 19:48:12.246375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.246349    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" event={"ID":"f128bd83-1027-45a5-b750-ce1920e34719","Type":"ContainerStarted","Data":"acec092ef69f0de5f9ef913252a01c6e2950c87167cf93ac531c93f2d1aebcc0"}
Oct 31 19:48:12.311566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.311385    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podStartSLOduration=66.311365747 podStartE2EDuration="1m6.311365747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:12.27946013 +0000 UTC m=+1685.740834746" watchObservedRunningTime="2026-10-31 19:48:12.311365747 +0000 UTC m=+1685.772740363"
Oct 31 19:48:12.316564 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.315735    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podStartSLOduration=55.315720974 podStartE2EDuration="55.315720974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:12.306689934 +0000 UTC m=+1685.768064544" watchObservedRunningTime="2026-10-31 19:48:12.315720974 +0000 UTC m=+1685.777095589"
Oct 31 19:48:12.334605 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.333380    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"]
Oct 31 19:48:12.342833 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.342758    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-6thpk"]
Oct 31 19:48:12.349933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.349871    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"]
Oct 31 19:48:12.357486 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.355773    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mfft9"]
Oct 31 19:48:12.673462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.673422    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7572f13d-f605-4362-96dc-0c1d5575833d" path="/var/lib/kubelet/pods/7572f13d-f605-4362-96dc-0c1d5575833d/volumes"
Oct 31 19:48:12.674289 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.674266    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9787d41b-dc62-49f6-83e8-51ab509f73f1" path="/var/lib/kubelet/pods/9787d41b-dc62-49f6-83e8-51ab509f73f1/volumes"
Oct 31 19:48:12.676868 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.676845    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:12.709987 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:12.709958    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:13.277102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:13.277069    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" event={"ID":"f128bd83-1027-45a5-b750-ce1920e34719","Type":"ContainerStarted","Data":"61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696"}
Oct 31 19:48:13.678652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:13.678612    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:48:13.678803 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:13.678670    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:48:13.732568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:13.729219    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:13.732568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:13.729288    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:14.676977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:14.676916    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:48:14.677506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:14.676997    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:48:14.719579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:14.718998    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:14.719579 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:14.719075    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:15.677467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:15.677430    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:48:15.677890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:15.677496    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:48:15.713705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:15.712897    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:15.713705 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:15.712958    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:16.677319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:16.677289    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:48:16.677506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:16.677337    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:48:16.726568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:16.725156    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:16.726568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:16.725219    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:17.678626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:17.677534    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:48:17.678626 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:17.677630    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:48:17.725570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:17.710328    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:17.725570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:17.710416    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:17.776954 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:48:17.776906    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:48:18.677787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:18.677755    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:48:18.678045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:18.677829    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 19:48:18.712938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:18.712899    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:18.713570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:18.713514    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:19.686476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:19.686443    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:19.687000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:19.686972    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:19.687866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:19.687839    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:48:19.710076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:19.710038    5958 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 31 19:48:19.710281 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:19.710106    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 31 19:48:20.716615 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:20.716580    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:20.717103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:20.717084    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:20.717808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:48:20.717785    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:48:47.048826 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:48:47.048742    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:49:08.766161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:08.766126    5958 scope.go:122] "RemoveContainer" containerID="12c20372e81bd12ab0fa000de0e3bfbffba24991d8086164ce79cc47a8009015"
Oct 31 19:49:16.039080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:16.039045    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"]
Oct 31 19:49:16.039478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:16.039319    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy" containerID="cri-o://c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b" gracePeriod=30
Oct 31 19:49:17.049439 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:49:17.049337    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:49:20.611603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:20.611567    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"]
Oct 31 19:49:20.612124 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:20.611939    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy" containerID="cri-o://ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed" gracePeriod=30
Oct 31 19:49:21.465618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.465591    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:49:21.527467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527425    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527488    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527597    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527666    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527828    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527949    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.527984    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.528056    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.528197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.528123    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.529060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.528208    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo\") pod \"9784b382-3851-48b5-bf8f-f176fea1d4b7\" (UID: \"9784b382-3851-48b5-bf8f-f176fea1d4b7\") "
Oct 31 19:49:21.529752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.529726    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.530037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.530005    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:21.530126 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.530087    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:21.530194 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.530137    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:21.530405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.530383    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:21.532871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.532808    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:21.532871 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.532847    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "kube-api-access-d458n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:21.535477 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.535439    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:49:21.535632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.535484    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy" pod "9784b382-3851-48b5-bf8f-f176fea1d4b7" (UID: "9784b382-3851-48b5-bf8f-f176fea1d4b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:21.631360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631324    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d458n\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-kube-api-access-d458n\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631353    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631360 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631362    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631373    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631385    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631398    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9784b382-3851-48b5-bf8f-f176fea1d4b7-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631409    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.631881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.631417    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9784b382-3851-48b5-bf8f-f176fea1d4b7-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:21.675340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.675308    5958 generic.go:356] "Generic (PLEG): container finished" podID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerID="c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b" exitCode=0
Oct 31 19:49:21.675508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.675357    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"
Oct 31 19:49:21.675508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.675387    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" event={"ID":"9784b382-3851-48b5-bf8f-f176fea1d4b7","Type":"ContainerDied","Data":"c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b"}
Oct 31 19:49:21.675508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.675439    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2" event={"ID":"9784b382-3851-48b5-bf8f-f176fea1d4b7","Type":"ContainerDied","Data":"e6801a06fd903ae193980f17552090e1cf5ab00f99f9e29040a8970ea0b18999"}
Oct 31 19:49:21.675508 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.675463    5958 scope.go:122] "RemoveContainer" containerID="c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b"
Oct 31 19:49:21.688978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.688955    5958 scope.go:122] "RemoveContainer" containerID="c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b"
Oct 31 19:49:21.689306 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:49:21.689278    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b\": container with ID starting with c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b not found: ID does not exist" containerID="c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b"
Oct 31 19:49:21.689389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.689313    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b"} err="failed to get container status \"c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b\": rpc error: code = NotFound desc = could not find container \"c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b\": container with ID starting with c2c682bae1779307694e5c669cef04d729dfd05aef1047e0306d7596e58d797b not found: ID does not exist"
Oct 31 19:49:21.706416 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.706357    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"]
Oct 31 19:49:21.711799 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:21.711768    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-mbvk2"]
Oct 31 19:49:22.668161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:22.668119    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" path="/var/lib/kubelet/pods/9784b382-3851-48b5-bf8f-f176fea1d4b7/volumes"
Oct 31 19:49:25.973044 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:25.973018    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:49:26.048496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.048463    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.048732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.048525    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.048732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.048690    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.048866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.048741    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.048866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.048834    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.048991 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.048884    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.049172 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049149    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.049280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049223    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.049398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049283    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.049398 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049386    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket\") pod \"74fcf135-c73f-4517-b815-b23a15bf4931\" (UID: \"74fcf135-c73f-4517-b815-b23a15bf4931\") "
Oct 31 19:49:26.049630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049602    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.049746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049701    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:26.049830 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049809    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.049942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.049923    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.052127 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052092    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.052732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052700    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:49:26.053016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052906    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.053016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052923    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.053016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052932    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.053016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052946    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74fcf135-c73f-4517-b815-b23a15bf4931-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.053016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052960    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.053016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.052969    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74fcf135-c73f-4517-b815-b23a15bf4931-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.054689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.054661    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "kube-api-access-pmrhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:26.054965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.054945    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token" pod "74fcf135-c73f-4517-b815-b23a15bf4931" (UID: "74fcf135-c73f-4517-b815-b23a15bf4931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:26.154530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.154489    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74fcf135-c73f-4517-b815-b23a15bf4931-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.154530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.154526    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrhs\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-kube-api-access-pmrhs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.154835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.154540    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74fcf135-c73f-4517-b815-b23a15bf4931-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.783251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.783216    5958 generic.go:356] "Generic (PLEG): container finished" podID="74fcf135-c73f-4517-b815-b23a15bf4931" containerID="ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed" exitCode=0
Oct 31 19:49:26.783464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.783282    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"
Oct 31 19:49:26.783464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.783307    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" event={"ID":"74fcf135-c73f-4517-b815-b23a15bf4931","Type":"ContainerDied","Data":"ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed"}
Oct 31 19:49:26.783464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.783368    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75" event={"ID":"74fcf135-c73f-4517-b815-b23a15bf4931","Type":"ContainerDied","Data":"172f07d90341de0fba376be7bf35c0dc82f8aac61ec8c26d843f5c6e20b828ec"}
Oct 31 19:49:26.783464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.783391    5958 scope.go:122] "RemoveContainer" containerID="ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed"
Oct 31 19:49:26.796746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.796722    5958 scope.go:122] "RemoveContainer" containerID="ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed"
Oct 31 19:49:26.797073 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:49:26.797049    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed\": container with ID starting with ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed not found: ID does not exist" containerID="ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed"
Oct 31 19:49:26.797174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.797081    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed"} err="failed to get container status \"ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed\": rpc error: code = NotFound desc = could not find container \"ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed\": container with ID starting with ea8a45923be9ed8390098795625b12d744fb4b00171a1240e7ea2a3b7be073ed not found: ID does not exist"
Oct 31 19:49:26.814065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.814036    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"]
Oct 31 19:49:26.819204 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:26.819181    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-r6b75"]
Oct 31 19:49:27.431901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:27.431867    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:49:28.679284 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:28.679247    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" path="/var/lib/kubelet/pods/74fcf135-c73f-4517-b815-b23a15bf4931/volumes"
Oct 31 19:49:47.048275 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:49:47.048233    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:49:54.345426 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:54.345377    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"]
Oct 31 19:49:54.363617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:54.363504    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"]
Oct 31 19:49:59.346603 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:59.346558    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerName="payload-processing" containerID="cri-o://cc62417c7dd942c7130724d7688613bfa7ff2b86c1ab4b64338f3a43b4b1f043" gracePeriod=25
Oct 31 19:49:59.364449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:49:59.364393    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" podUID="280ebe12-7060-4a50-885a-c20765df1460" containerName="payload-pre-processing" containerID="cri-o://1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01" gracePeriod=25
Oct 31 19:50:05.685919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:05.685880    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"]
Oct 31 19:50:05.686343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:05.686101    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" containerID="cri-o://a9f5cefb70143207fae5142fb03152ebba0228395ae244f97f6535486e13b2ed" gracePeriod=30
Oct 31 19:50:06.161643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.160353    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"]
Oct 31 19:50:06.161643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.160723    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy" containerID="cri-o://bc0afd3efeff1efdd2e9541c50705130d312246c02cf88162aab96168769cf5c" gracePeriod=30
Oct 31 19:50:06.169102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.169045    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:06.184688 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.171657    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" podUID="280ebe12-7060-4a50-885a-c20765df1460" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:06.192622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.188865    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"]
Oct 31 19:50:06.192622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.189125    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" containerID="cri-o://e2a5d2a0b4b0c02a7655a98835ee6a7ff633ca6317bf7666aad14bfd2127ca99" gracePeriod=30
Oct 31 19:50:06.210185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.201890    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"]
Oct 31 19:50:06.210185 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:06.202154    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy" containerID="cri-o://b959a199a64da8d68195b01302234b5084925a1f24269b792ffa8b846de6b67e" gracePeriod=30
Oct 31 19:50:08.724390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:08.724359    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"]
Oct 31 19:50:08.724822 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:08.724571    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy" containerID="cri-o://cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6" gracePeriod=30
Oct 31 19:50:12.121566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.119442    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 19:50:12.121566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.119520    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 19:50:12.299154 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.298794    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.326247    5958 generic.go:356] "Generic (PLEG): container finished" podID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerID="bc0afd3efeff1efdd2e9541c50705130d312246c02cf88162aab96168769cf5c" exitCode=0
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.326580    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" event={"ID":"a5f705aa-c316-4b3f-a179-9f65434398c3","Type":"ContainerDied","Data":"bc0afd3efeff1efdd2e9541c50705130d312246c02cf88162aab96168769cf5c"}
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.329897    5958 generic.go:356] "Generic (PLEG): container finished" podID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerID="e2a5d2a0b4b0c02a7655a98835ee6a7ff633ca6317bf7666aad14bfd2127ca99" exitCode=0
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.330008    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" event={"ID":"9fdca189-5947-4c03-a1f8-790bf8c6a68d","Type":"ContainerDied","Data":"e2a5d2a0b4b0c02a7655a98835ee6a7ff633ca6317bf7666aad14bfd2127ca99"}
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.333029    5958 generic.go:356] "Generic (PLEG): container finished" podID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerID="a9f5cefb70143207fae5142fb03152ebba0228395ae244f97f6535486e13b2ed" exitCode=0
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.333320    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" event={"ID":"5b10670d-0976-4783-be38-d7dc50ee95b4","Type":"ContainerDied","Data":"a9f5cefb70143207fae5142fb03152ebba0228395ae244f97f6535486e13b2ed"}
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.333391    5958 scope.go:122] "RemoveContainer" containerID="a9f5cefb70143207fae5142fb03152ebba0228395ae244f97f6535486e13b2ed"
Oct 31 19:50:12.345572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.333500    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"
Oct 31 19:50:12.356862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.356723    5958 generic.go:356] "Generic (PLEG): container finished" podID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerID="b959a199a64da8d68195b01302234b5084925a1f24269b792ffa8b846de6b67e" exitCode=0
Oct 31 19:50:12.357064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.356958    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" event={"ID":"9c9469d9-55d6-4932-89e6-23a41a08e600","Type":"ContainerDied","Data":"b959a199a64da8d68195b01302234b5084925a1f24269b792ffa8b846de6b67e"}
Oct 31 19:50:12.414739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.414364    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.421889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.421855    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.422135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.422118    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.422288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.422276    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.422433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.422422    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.422584 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.422571    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.422899 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.422885    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.440893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.440863    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.477411 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.477371    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9\") pod \"5b10670d-0976-4783-be38-d7dc50ee95b4\" (UID: \"5b10670d-0976-4783-be38-d7dc50ee95b4\") "
Oct 31 19:50:12.509271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.486030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.509271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.494982    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.509271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.495379    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:12.509271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.495664    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.509271 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.505684    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.533140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.531813    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.533140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.531851    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b10670d-0976-4783-be38-d7dc50ee95b4-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.533140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.531867    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.533140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.531884    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.533140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.531898    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.593569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.588315    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.609393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.593814    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:50:12.609393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.594648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "kube-api-access-7qvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:12.628082 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.626527    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:12.628569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.628321    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 31 19:50:12.647911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.634767    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvj9\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-kube-api-access-7qvj9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.708663 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.708584    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.708890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.708869    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.709006 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.706925    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token" pod "5b10670d-0976-4783-be38-d7dc50ee95b4" (UID: "5b10670d-0976-4783-be38-d7dc50ee95b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:12.770192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.769846    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:12.770192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.769921    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:12.771650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.771617    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:12.812987 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.812944    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.814632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.814598    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.814892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.814869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.815184 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.815163    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.815372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.815353    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.815569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.815537    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.815744 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.815728    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.815919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.815904    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.816059 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.816044    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token\") pod \"a5f705aa-c316-4b3f-a179-9f65434398c3\" (UID: \"a5f705aa-c316-4b3f-a179-9f65434398c3\") "
Oct 31 19:50:12.817413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.817389    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b10670d-0976-4783-be38-d7dc50ee95b4-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.818229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.818199    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.826806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.826768    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.827330 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.827125    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:12.827587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.827561    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.829029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.829003    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.829683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.829438    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "kube-api-access-jsrlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:12.835019 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.834991    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:12.847978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.844682    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.851023 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.850660    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token" pod "a5f705aa-c316-4b3f-a179-9f65434398c3" (UID: "a5f705aa-c316-4b3f-a179-9f65434398c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:12.919874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919799    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919901    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919919    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919933    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919953    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919964    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919976    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f705aa-c316-4b3f-a179-9f65434398c3-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.919991    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrlq\" (UniqueName: \"kubernetes.io/projected/a5f705aa-c316-4b3f-a179-9f65434398c3-kube-api-access-jsrlq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.920052 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.920006    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f705aa-c316-4b3f-a179-9f65434398c3-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.921391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.921364    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:50:12.924669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.924643    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:50:12.977728 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.977629    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"]
Oct 31 19:50:12.982227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:12.982189    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-mbv4q"]
Oct 31 19:50:13.020817 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.020771    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.020839    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.020866    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.020991    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021022    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021046    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021082    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021101    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021177    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021210 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021197    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021230    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021303    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021355    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021405    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021517 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021452    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021523    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo\") pod \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\" (UID: \"9fdca189-5947-4c03-a1f8-790bf8c6a68d\") "
Oct 31 19:50:13.021831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021555    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.021831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.021572    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket\") pod \"9c9469d9-55d6-4932-89e6-23a41a08e600\" (UID: \"9c9469d9-55d6-4932-89e6-23a41a08e600\") "
Oct 31 19:50:13.022532 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.022498    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.023484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.023230    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.024889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.024192    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.024889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.024371    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.024889 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.024648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:13.025673 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.025370    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.026064 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.026037    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "kube-api-access-zgflc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.026268    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.026566    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.026809    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.026989    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.028968    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.029857    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "kube-api-access-dfr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.029974    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.030087    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.030177    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token" pod "9c9469d9-55d6-4932-89e6-23a41a08e600" (UID: "9c9469d9-55d6-4932-89e6-23a41a08e600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:13.031672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.031049    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:13.032241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.031859    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy" pod "9fdca189-5947-4c03-a1f8-790bf8c6a68d" (UID: "9fdca189-5947-4c03-a1f8-790bf8c6a68d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:13.123703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123663    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.123703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123693    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.123703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123703    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.123703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123712    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123723    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9469d9-55d6-4932-89e6-23a41a08e600-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123733    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgflc\" (UniqueName: \"kubernetes.io/projected/9fdca189-5947-4c03-a1f8-790bf8c6a68d-kube-api-access-zgflc\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123743    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123752    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123761    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123770    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123778    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123786    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-kube-api-access-dfr4s\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123794    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123803    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123812    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9469d9-55d6-4932-89e6-23a41a08e600-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123820    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9469d9-55d6-4932-89e6-23a41a08e600-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123830    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.124339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.123837    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fdca189-5947-4c03-a1f8-790bf8c6a68d-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.367443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.367411    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"
Oct 31 19:50:13.367686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.367445    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr" event={"ID":"a5f705aa-c316-4b3f-a179-9f65434398c3","Type":"ContainerDied","Data":"2ae02c18f127dd6840789dec41edc280ebdde74982eb17b1719dc41d02032028"}
Oct 31 19:50:13.367686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.367509    5958 scope.go:122] "RemoveContainer" containerID="bc0afd3efeff1efdd2e9541c50705130d312246c02cf88162aab96168769cf5c"
Oct 31 19:50:13.369308 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.369281    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj" event={"ID":"9fdca189-5947-4c03-a1f8-790bf8c6a68d","Type":"ContainerDied","Data":"a356cadb44d63d7a05c1ebf61546fa61750d91fc09d5231b7cdb2acb6d7e53c5"}
Oct 31 19:50:13.369496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.369290    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"
Oct 31 19:50:13.372348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.372317    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"
Oct 31 19:50:13.372465 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.372355    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw" event={"ID":"9c9469d9-55d6-4932-89e6-23a41a08e600","Type":"ContainerDied","Data":"f9463ea97ab90755bfdd4c4941166edb13b8f80dd8896966cd2b4221de426abe"}
Oct 31 19:50:13.387827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.387369    5958 scope.go:122] "RemoveContainer" containerID="e2a5d2a0b4b0c02a7655a98835ee6a7ff633ca6317bf7666aad14bfd2127ca99"
Oct 31 19:50:13.399152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.399132    5958 scope.go:122] "RemoveContainer" containerID="b959a199a64da8d68195b01302234b5084925a1f24269b792ffa8b846de6b67e"
Oct 31 19:50:13.408806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.408754    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"]
Oct 31 19:50:13.415601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.415576    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-mq4hj"]
Oct 31 19:50:13.435994 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.435970    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"]
Oct 31 19:50:13.440683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.440657    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-88shr"]
Oct 31 19:50:13.463632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.463602    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"]
Oct 31 19:50:13.477143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:13.474496    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-s75bw"]
Oct 31 19:50:14.043958 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.043931    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:50:14.093713 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.093681    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.093906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.093859    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.094966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.094928    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:14.095277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.095255    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.095400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.095302    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.095400 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.095387    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.095645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.095492    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.095645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.095592    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.096345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.095998    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.096345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.096054    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy\") pod \"1a739028-c5f1-4fc2-98fb-68800d3269f3\" (UID: \"1a739028-c5f1-4fc2-98fb-68800d3269f3\") "
Oct 31 19:50:14.097106 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.097052    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a739028-c5f1-4fc2-98fb-68800d3269f3-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.097492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.097462    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:14.098647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.097826    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:14.098847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.098819    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:14.099093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.099042    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:14.099497 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.099462    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:14.106759 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.106720    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:14.106943 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.106917    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "kube-api-access-7wwzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:14.107065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.106981    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy" pod "1a739028-c5f1-4fc2-98fb-68800d3269f3" (UID: "1a739028-c5f1-4fc2-98fb-68800d3269f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218289    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218329    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218347    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218361    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218374    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218387    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218400    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a739028-c5f1-4fc2-98fb-68800d3269f3-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.225629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.218413    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwzd\" (UniqueName: \"kubernetes.io/projected/1a739028-c5f1-4fc2-98fb-68800d3269f3-kube-api-access-7wwzd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.382140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.382107    5958 generic.go:356] "Generic (PLEG): container finished" podID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerID="cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6" exitCode=0
Oct 31 19:50:14.382343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.382153    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"
Oct 31 19:50:14.382343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.382182    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" event={"ID":"1a739028-c5f1-4fc2-98fb-68800d3269f3","Type":"ContainerDied","Data":"cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6"}
Oct 31 19:50:14.382343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.382243    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf" event={"ID":"1a739028-c5f1-4fc2-98fb-68800d3269f3","Type":"ContainerDied","Data":"4ea5adcfb9a45d69ccb5e771ff867b07a0b58b62d724f7cb892129482e017298"}
Oct 31 19:50:14.382343 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.382276    5958 scope.go:122] "RemoveContainer" containerID="cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6"
Oct 31 19:50:14.399510 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.399491    5958 scope.go:122] "RemoveContainer" containerID="cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6"
Oct 31 19:50:14.399816 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:14.399795    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6\": container with ID starting with cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6 not found: ID does not exist" containerID="cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6"
Oct 31 19:50:14.399887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.399826    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6"} err="failed to get container status \"cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6\": rpc error: code = NotFound desc = could not find container \"cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6\": container with ID starting with cb0f378b4d11f3a04145369135de380bdcefbf25c14bf037290f4fe5c13debf6 not found: ID does not exist"
Oct 31 19:50:14.416631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.416600    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"]
Oct 31 19:50:14.419040 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.419019    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-4fnkf"]
Oct 31 19:50:14.668462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.668379    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" path="/var/lib/kubelet/pods/1a739028-c5f1-4fc2-98fb-68800d3269f3/volumes"
Oct 31 19:50:14.669120 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.669094    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" path="/var/lib/kubelet/pods/5b10670d-0976-4783-be38-d7dc50ee95b4/volumes"
Oct 31 19:50:14.669687 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.669671    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" path="/var/lib/kubelet/pods/9c9469d9-55d6-4932-89e6-23a41a08e600/volumes"
Oct 31 19:50:14.670485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.670469    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" path="/var/lib/kubelet/pods/9fdca189-5947-4c03-a1f8-790bf8c6a68d/volumes"
Oct 31 19:50:14.670991 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:14.670971    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" path="/var/lib/kubelet/pods/a5f705aa-c316-4b3f-a179-9f65434398c3/volumes"
Oct 31 19:50:16.042076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:16.042028    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:16.042583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:16.042319    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" podUID="280ebe12-7060-4a50-885a-c20765df1460" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:17.049038 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:17.049000    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:50:19.470133 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:19.470099    5958 generic.go:356] "Generic (PLEG): container finished" podID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerID="cc62417c7dd942c7130724d7688613bfa7ff2b86c1ab4b64338f3a43b4b1f043" exitCode=0
Oct 31 19:50:19.470651 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:19.470164    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" event={"ID":"65bf01c1-da58-4383-a303-8ffc1ea1289b","Type":"ContainerDied","Data":"cc62417c7dd942c7130724d7688613bfa7ff2b86c1ab4b64338f3a43b4b1f043"}
Oct 31 19:50:20.203818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.199752    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:50:20.330652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.326344    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z\" (UniqueName: \"kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z\") pod \"65bf01c1-da58-4383-a303-8ffc1ea1289b\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") "
Oct 31 19:50:20.330652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.326499    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume\") pod \"65bf01c1-da58-4383-a303-8ffc1ea1289b\" (UID: \"65bf01c1-da58-4383-a303-8ffc1ea1289b\") "
Oct 31 19:50:20.330652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.329469    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume" pod "65bf01c1-da58-4383-a303-8ffc1ea1289b" (UID: "65bf01c1-da58-4383-a303-8ffc1ea1289b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:20.339228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.339159    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z" pod "65bf01c1-da58-4383-a303-8ffc1ea1289b" (UID: "65bf01c1-da58-4383-a303-8ffc1ea1289b"). InnerVolumeSpecName "kube-api-access-xks5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:20.420057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.420026    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:50:20.430214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.430160    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xks5z\" (UniqueName: \"kubernetes.io/projected/65bf01c1-da58-4383-a303-8ffc1ea1289b-kube-api-access-xks5z\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.430387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.430281    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65bf01c1-da58-4383-a303-8ffc1ea1289b-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.500121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.500080    5958 generic.go:356] "Generic (PLEG): container finished" podID="280ebe12-7060-4a50-885a-c20765df1460" containerID="1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01" exitCode=0
Oct 31 19:50:20.500526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.500231    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" event={"ID":"280ebe12-7060-4a50-885a-c20765df1460","Type":"ContainerDied","Data":"1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01"}
Oct 31 19:50:20.500526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.500289    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk" event={"ID":"280ebe12-7060-4a50-885a-c20765df1460","Type":"ContainerDied","Data":"f3553d936ac2a417101202bcc0b606183ead5b3e88902999eee4d6e165c5b573"}
Oct 31 19:50:20.500526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.500322    5958 scope.go:122] "RemoveContainer" containerID="1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01"
Oct 31 19:50:20.500526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.500442    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"
Oct 31 19:50:20.503298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.503266    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v" event={"ID":"65bf01c1-da58-4383-a303-8ffc1ea1289b","Type":"ContainerDied","Data":"3bb663e90e7b252d19221d03101db98c50d8ecb455cf8e99dadbf988d4862d99"}
Oct 31 19:50:20.503451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.503381    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"
Oct 31 19:50:20.520785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.520757    5958 scope.go:122] "RemoveContainer" containerID="1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01"
Oct 31 19:50:20.521129 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:20.521106    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01\": container with ID starting with 1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01 not found: ID does not exist" containerID="1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01"
Oct 31 19:50:20.521242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.521139    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01"} err="failed to get container status \"1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01\": rpc error: code = NotFound desc = could not find container \"1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01\": container with ID starting with 1257dac5f82e690fce48c9bec51060af63613bc078d068985db0a7f1c7fabb01 not found: ID does not exist"
Oct 31 19:50:20.521242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.521158    5958 scope.go:122] "RemoveContainer" containerID="cc62417c7dd942c7130724d7688613bfa7ff2b86c1ab4b64338f3a43b4b1f043"
Oct 31 19:50:20.532046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.532020    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume\") pod \"280ebe12-7060-4a50-885a-c20765df1460\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") "
Oct 31 19:50:20.532158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.532079    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt\" (UniqueName: \"kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt\") pod \"280ebe12-7060-4a50-885a-c20765df1460\" (UID: \"280ebe12-7060-4a50-885a-c20765df1460\") "
Oct 31 19:50:20.533183 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.533155    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume" pod "280ebe12-7060-4a50-885a-c20765df1460" (UID: "280ebe12-7060-4a50-885a-c20765df1460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:20.535534 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.535499    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt" pod "280ebe12-7060-4a50-885a-c20765df1460" (UID: "280ebe12-7060-4a50-885a-c20765df1460"). InnerVolumeSpecName "kube-api-access-6gtzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:20.555730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.543136    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"]
Oct 31 19:50:20.566966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.566919    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-03dfee-6fb7d4f559-mhj7v"]
Oct 31 19:50:20.637926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.633973    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gtzt\" (UniqueName: \"kubernetes.io/projected/280ebe12-7060-4a50-885a-c20765df1460-kube-api-access-6gtzt\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.637926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.634007    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/280ebe12-7060-4a50-885a-c20765df1460-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.671098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.671058    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" path="/var/lib/kubelet/pods/65bf01c1-da58-4383-a303-8ffc1ea1289b/volumes"
Oct 31 19:50:20.828337 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.828292    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"]
Oct 31 19:50:20.833528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:20.833495    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-03dfee-fc9fd5b8c-p6lgk"]
Oct 31 19:50:22.668077 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:22.668042    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280ebe12-7060-4a50-885a-c20765df1460" path="/var/lib/kubelet/pods/280ebe12-7060-4a50-885a-c20765df1460/volumes"
Oct 31 19:50:32.031961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:32.031920    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"]
Oct 31 19:50:32.058632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:32.057321    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"]
Oct 31 19:50:37.033032 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.032984    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w" podUID="41571f05-1414-4dbb-9f85-5505daae3307" containerName="payload-processing" containerID="cri-o://10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb" gracePeriod=25
Oct 31 19:50:37.058632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.058595    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerName="payload-pre-processing" containerID="cri-o://3b6bb0d11923e7a410828e0eca6e16fb62fc0dbdcacb247fbcc9d37ef86194bd" gracePeriod=25
Oct 31 19:50:37.365680 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.365654    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:50:37.539953 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.539921    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume\") pod \"41571f05-1414-4dbb-9f85-5505daae3307\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") "
Oct 31 19:50:37.540207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.540194    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286\" (UniqueName: \"kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286\") pod \"41571f05-1414-4dbb-9f85-5505daae3307\" (UID: \"41571f05-1414-4dbb-9f85-5505daae3307\") "
Oct 31 19:50:37.540369 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.540342    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume" pod "41571f05-1414-4dbb-9f85-5505daae3307" (UID: "41571f05-1414-4dbb-9f85-5505daae3307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:37.540883 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.540860    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41571f05-1414-4dbb-9f85-5505daae3307-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.542306 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.542275    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286" pod "41571f05-1414-4dbb-9f85-5505daae3307" (UID: "41571f05-1414-4dbb-9f85-5505daae3307"). InnerVolumeSpecName "kube-api-access-lh286". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:37.643861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.642523    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh286\" (UniqueName: \"kubernetes.io/projected/41571f05-1414-4dbb-9f85-5505daae3307-kube-api-access-lh286\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.945617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.944892    5958 generic.go:356] "Generic (PLEG): container finished" podID="41571f05-1414-4dbb-9f85-5505daae3307" containerID="10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb" exitCode=0
Oct 31 19:50:37.945617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.945200    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w" event={"ID":"41571f05-1414-4dbb-9f85-5505daae3307","Type":"ContainerDied","Data":"10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb"}
Oct 31 19:50:37.945617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.945256    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w" event={"ID":"41571f05-1414-4dbb-9f85-5505daae3307","Type":"ContainerDied","Data":"f949df8a55bd3170f4a48e19117fe14336379e54d15affe1033bd2d0799f45ce"}
Oct 31 19:50:37.945617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.945290    5958 scope.go:122] "RemoveContainer" containerID="10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb"
Oct 31 19:50:37.954516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:37.954479    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"
Oct 31 19:50:38.010672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:38.010647    5958 scope.go:122] "RemoveContainer" containerID="10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb"
Oct 31 19:50:38.011124 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:38.011093    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb\": container with ID starting with 10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb not found: ID does not exist" containerID="10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb"
Oct 31 19:50:38.011179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:38.011153    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb"} err="failed to get container status \"10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb\": rpc error: code = NotFound desc = could not find container \"10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb\": container with ID starting with 10b7b5a8f1a1555b75cd1eec294796e3fa45a3df787a97e95d21a42a62ffb0eb not found: ID does not exist"
Oct 31 19:50:38.033553 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:38.033500    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"]
Oct 31 19:50:38.038984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:38.038416    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1fae7-6c4988f4d5-x9g5w"]
Oct 31 19:50:38.668512 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:38.668481    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41571f05-1414-4dbb-9f85-5505daae3307" path="/var/lib/kubelet/pods/41571f05-1414-4dbb-9f85-5505daae3307/volumes"
Oct 31 19:50:40.466039 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.465994    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"]
Oct 31 19:50:40.466557 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.466220    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" podUID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" containerName="maas-api" containerID="cri-o://d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a" gracePeriod=30
Oct 31 19:50:40.586959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.586918    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" podUID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.168:8443/health\": dial tcp 10.134.0.168:8443: connect: connection refused"
Oct 31 19:50:40.882274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.882245    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:50:40.960103 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.960060    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8\" (UniqueName: \"kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8\") pod \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") "
Oct 31 19:50:40.960287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.960148    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls\") pod \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") "
Oct 31 19:50:40.960287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.960184    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca\") pod \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") "
Oct 31 19:50:40.960287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.960254    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs\") pod \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\" (UID: \"cbe49af0-be0b-4e31-b75b-5ca0ab985bee\") "
Oct 31 19:50:40.960968 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.960829    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca" pod "cbe49af0-be0b-4e31-b75b-5ca0ab985bee" (UID: "cbe49af0-be0b-4e31-b75b-5ca0ab985bee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:40.961128 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.961104    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:40.962842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.962811    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls" pod "cbe49af0-be0b-4e31-b75b-5ca0ab985bee" (UID: "cbe49af0-be0b-4e31-b75b-5ca0ab985bee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:40.962965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.962872    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8" pod "cbe49af0-be0b-4e31-b75b-5ca0ab985bee" (UID: "cbe49af0-be0b-4e31-b75b-5ca0ab985bee"). InnerVolumeSpecName "kube-api-access-hrwc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:40.963055 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.963034    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs" pod "cbe49af0-be0b-4e31-b75b-5ca0ab985bee" (UID: "cbe49af0-be0b-4e31-b75b-5ca0ab985bee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:40.976349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.976256    5958 generic.go:356] "Generic (PLEG): container finished" podID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" containerID="d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a" exitCode=0
Oct 31 19:50:40.976349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.976315    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"
Oct 31 19:50:40.976349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.976336    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" event={"ID":"cbe49af0-be0b-4e31-b75b-5ca0ab985bee","Type":"ContainerDied","Data":"d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a"}
Oct 31 19:50:40.976672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.976388    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx" event={"ID":"cbe49af0-be0b-4e31-b75b-5ca0ab985bee","Type":"ContainerDied","Data":"419639c65bee4ecbd8bdb1417952adc5230c90d49ef9190f14c32deb7287e9d9"}
Oct 31 19:50:40.976672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:40.976415    5958 scope.go:122] "RemoveContainer" containerID="d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a"
Oct 31 19:50:41.020420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.020395    5958 scope.go:122] "RemoveContainer" containerID="d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a"
Oct 31 19:50:41.020810 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:41.020788    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a\": container with ID starting with d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a not found: ID does not exist" containerID="d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a"
Oct 31 19:50:41.020918 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.020822    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a"} err="failed to get container status \"d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a\": rpc error: code = NotFound desc = could not find container \"d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a\": container with ID starting with d2a826b46672f36d127f2c629b6c0573fee5d80e62d84b0b1b66aca53bbc687a not found: ID does not exist"
Oct 31 19:50:41.049526 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.049493    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"]
Oct 31 19:50:41.059116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.059087    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b1fae7-69f7649488-lw5sx"]
Oct 31 19:50:41.062809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.062777    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwc8\" (UniqueName: \"kubernetes.io/projected/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-kube-api-access-hrwc8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:41.062901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.062813    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:41.062901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:41.062831    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe49af0-be0b-4e31-b75b-5ca0ab985bee-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:42.667691 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:42.667662    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" path="/var/lib/kubelet/pods/cbe49af0-be0b-4e31-b75b-5ca0ab985bee/volumes"
Oct 31 19:50:45.470799 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:45.470756    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"]
Oct 31 19:50:45.471696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:45.471661    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy" containerID="cri-o://601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73" gracePeriod=30
Oct 31 19:50:45.481643 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:45.481499    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"]
Oct 31 19:50:45.481787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:45.481761    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" containerID="cri-o://fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1" gracePeriod=30
Oct 31 19:50:46.740692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:46.740645    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:47.050616 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:47.050508    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:50:48.478487 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:48.478452    5958 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:50:48.478946 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:48.478510    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:50:51.079080 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.044797    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:50:51.126118 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.126089    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:50:51.133319 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133181    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.133469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133316    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.133469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133351    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.133469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133371    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.133469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133424    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.135406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133767    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.135406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133904    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.135406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.133988    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.135406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.134043    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy\") pod \"a619cde9-79d6-40f4-a796-91f16966cc8a\" (UID: \"a619cde9-79d6-40f4-a796-91f16966cc8a\") "
Oct 31 19:50:51.135784 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.135705    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.135916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.135896    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.138921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.138894    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.139287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.139263    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.141656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.141608    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.141762 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.141662    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:51.144389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.143985    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.144389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.144159    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "kube-api-access-gpx4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.146427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.146400    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo" pod "a619cde9-79d6-40f4-a796-91f16966cc8a" (UID: "a619cde9-79d6-40f4-a796-91f16966cc8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:51.146576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.146498    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"]
Oct 31 19:50:51.146937 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.146907    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" containerID="cri-o://637dcbba0a35eceaeb3e4b57cfdf6b216f21f82335ede4f82189cb740e151090" gracePeriod=30
Oct 31 19:50:51.235992 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.235955    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.236242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236111    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.236242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236225    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236253    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236277    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236309    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236574    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236651    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.236865    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.237211    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.237591    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.237922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.237772    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token\") pod \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\" (UID: \"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0\") "
Oct 31 19:50:51.238514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238073    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238787    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238813    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238829    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpx4n\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-kube-api-access-gpx4n\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238846    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238862    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238880    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238893    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238907    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238920    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238933    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619cde9-79d6-40f4-a796-91f16966cc8a-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238946    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238960    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619cde9-79d6-40f4-a796-91f16966cc8a-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.239135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.238973    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619cde9-79d6-40f4-a796-91f16966cc8a-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.240042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.239322    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:51.240937 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.240906    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.241175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.241149    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "kube-api-access-xn5n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.241286 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.241189    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:51.242198 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.242178    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token" pod "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" (UID: "32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.292402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.292187    5958 generic.go:356] "Generic (PLEG): container finished" podID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerID="fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1" exitCode=0
Oct 31 19:50:51.292402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.292257    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"
Oct 31 19:50:51.292402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.292289    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" event={"ID":"a619cde9-79d6-40f4-a796-91f16966cc8a","Type":"ContainerDied","Data":"fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1"}
Oct 31 19:50:51.292402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.292362    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs" event={"ID":"a619cde9-79d6-40f4-a796-91f16966cc8a","Type":"ContainerDied","Data":"bf2323749b8ee4b248e84eb64f4a2a1b451efa4539a1fbbc9158922ecdda99af"}
Oct 31 19:50:51.292402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.292393    5958 scope.go:122] "RemoveContainer" containerID="fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1"
Oct 31 19:50:51.294594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.294531    5958 generic.go:356] "Generic (PLEG): container finished" podID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerID="601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73" exitCode=0
Oct 31 19:50:51.294777 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.294757    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" event={"ID":"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0","Type":"ContainerDied","Data":"601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73"}
Oct 31 19:50:51.294862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.294810    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v" event={"ID":"32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0","Type":"ContainerDied","Data":"9750d42a68ddabd29ab04116685effa912adad001e3f23f65a4f6b8df2310924"}
Oct 31 19:50:51.294926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.294889    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"
Oct 31 19:50:51.315773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.315749    5958 scope.go:122] "RemoveContainer" containerID="fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1"
Oct 31 19:50:51.316442 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:51.316418    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1\": container with ID starting with fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1 not found: ID does not exist" containerID="fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1"
Oct 31 19:50:51.316948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.316452    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1"} err="failed to get container status \"fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1\": rpc error: code = NotFound desc = could not find container \"fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1\": container with ID starting with fa10813b772170ba2721947a4fa3eeeaf945ecb73c1b389ae7e0a23d29b025a1 not found: ID does not exist"
Oct 31 19:50:51.316948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.316478    5958 scope.go:122] "RemoveContainer" containerID="601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73"
Oct 31 19:50:51.329121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.329101    5958 scope.go:122] "RemoveContainer" containerID="601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73"
Oct 31 19:50:51.329445 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:51.329424    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73\": container with ID starting with 601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73 not found: ID does not exist" containerID="601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73"
Oct 31 19:50:51.329494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.329455    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73"} err="failed to get container status \"601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73\": rpc error: code = NotFound desc = could not find container \"601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73\": container with ID starting with 601d42a3e3b103540afbd3541f84529be564e8d8e8095d0ab971e09cf1ecde73 not found: ID does not exist"
Oct 31 19:50:51.340634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.340609    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5n8\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-kube-api-access-xn5n8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.340634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.340636    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.340832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.340646    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.340832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.340657    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.340832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.340666    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.350842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.350809    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"]
Oct 31 19:50:51.359168 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.359142    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-hqf9v"]
Oct 31 19:50:51.391907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.391830    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"]
Oct 31 19:50:51.410232 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.410190    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-rkpxs"]
Oct 31 19:50:51.555266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.555226    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:50:51.555427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:51.555296    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:50:52.670795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:52.670758    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" path="/var/lib/kubelet/pods/32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0/volumes"
Oct 31 19:50:52.671781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:52.671754    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" path="/var/lib/kubelet/pods/a619cde9-79d6-40f4-a796-91f16966cc8a/volumes"
Oct 31 19:50:52.755572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:52.755529    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"]
Oct 31 19:50:52.755802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:52.755778    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy" containerID="cri-o://61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696" gracePeriod=30
Oct 31 19:50:52.760084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:52.760057    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"]
Oct 31 19:50:52.760273 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:52.760252    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy" containerID="cri-o://7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d" gracePeriod=30
Oct 31 19:50:56.311881 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.311847    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"]
Oct 31 19:50:56.312780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312764    5958 cpu_manager.go:414] "containerMap: removing container" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312781    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312798    5958 cpu_manager.go:414] "containerMap: removing container" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312803    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312820    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312825    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312845    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312850    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312857    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy"
Oct 31 19:50:56.312861 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312862    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312878    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312882    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312904    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312909    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312918    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312923    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312929    5958 cpu_manager.go:414] "containerMap: removing container" podUID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" containerName="maas-api"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312934    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" containerName="maas-api"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312941    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312946    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312953    5958 cpu_manager.go:414] "containerMap: removing container" podUID="41571f05-1414-4dbb-9f85-5505daae3307" containerName="payload-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312957    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41571f05-1414-4dbb-9f85-5505daae3307" containerName="payload-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312977    5958 cpu_manager.go:414] "containerMap: removing container" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerName="payload-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312981    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerName="payload-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312988    5958 cpu_manager.go:414] "containerMap: removing container" podUID="280ebe12-7060-4a50-885a-c20765df1460" containerName="payload-pre-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.312992    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ebe12-7060-4a50-885a-c20765df1460" containerName="payload-pre-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313122    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f705aa-c316-4b3f-a179-9f65434398c3" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313133    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fdca189-5947-4c03-a1f8-790bf8c6a68d" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313143    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9469d9-55d6-4932-89e6-23a41a08e600" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313151    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe49af0-be0b-4e31-b75b-5ca0ab985bee" containerName="maas-api"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313157    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9784b382-3851-48b5-bf8f-f176fea1d4b7" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313163    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="41571f05-1414-4dbb-9f85-5505daae3307" containerName="payload-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313171    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="65bf01c1-da58-4383-a303-8ffc1ea1289b" containerName="payload-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313176    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ebe12-7060-4a50-885a-c20765df1460" containerName="payload-pre-processing"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313181    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fcf135-c73f-4517-b815-b23a15bf4931" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313188    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b1eb01-325e-4df9-9e5a-4ef67eb8c1c0" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313196    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a619cde9-79d6-40f4-a796-91f16966cc8a" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313204    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b10670d-0976-4783-be38-d7dc50ee95b4" containerName="istio-proxy"
Oct 31 19:50:56.313406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.313210    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a739028-c5f1-4fc2-98fb-68800d3269f3" containerName="istio-proxy"
Oct 31 19:50:56.317891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.317871    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.325695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.325642    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c55cf1\""
Oct 31 19:50:56.326808 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.326785    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c55cf1-dockercfg-8br6r\""
Oct 31 19:50:56.330163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.330137    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"]
Oct 31 19:50:56.350206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.350120    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerID="637dcbba0a35eceaeb3e4b57cfdf6b216f21f82335ede4f82189cb740e151090" exitCode=0
Oct 31 19:50:56.350206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.350194    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" event={"ID":"fc4a7691-ea30-4bca-bc1b-0b195ef17a00","Type":"ContainerDied","Data":"637dcbba0a35eceaeb3e4b57cfdf6b216f21f82335ede4f82189cb740e151090"}
Oct 31 19:50:56.569735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.525642    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9brf\" (UniqueName: \"kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf\") pod \"payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.569735 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.525803    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.621034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.619633    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"]
Oct 31 19:50:56.651163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.642063    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9brf\" (UniqueName: \"kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf\") pod \"payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.651163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.642270    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.651163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.649478    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.672427 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.672381    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.674242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.674203    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9brf\" (UniqueName: \"kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf\") pod \"payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.746725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.746681    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpbn\" (UniqueName: \"kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn\") pod \"payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.747063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.747037    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.754746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.754709    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:56.756297 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.756263    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"]
Oct 31 19:50:56.764845 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.764814    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"]
Oct 31 19:50:56.775893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.775862    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"]
Oct 31 19:50:56.782985 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.782962    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:50:56.849638 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.849504    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.850295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.850243    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpbn\" (UniqueName: \"kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn\") pod \"payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.850531 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.850506    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.868443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.868410    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpbn\" (UniqueName: \"kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn\") pod \"payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:56.944852 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.944809    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:56.951690 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951663    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.951821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951740    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.951821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951781    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.951949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951874    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.951949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951891    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.951949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951908    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.952105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.951977    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.952105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952052    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.952105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952081    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:56.952266 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952162    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy\") pod \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\" (UID: \"fc4a7691-ea30-4bca-bc1b-0b195ef17a00\") "
Oct 31 19:50:56.952355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952315    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:56.952699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952352    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:56.952841 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952697    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:56.952954 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.952927    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:56.953901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.953344    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.953901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.953390    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.953901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.953406    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.953901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.953420    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.953901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.953436    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.960349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.954533    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:56.960349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.954641    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:56.960349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.954843    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "kube-api-access-7qsnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:56.960349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:56.956659    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy" pod "fc4a7691-ea30-4bca-bc1b-0b195ef17a00" (UID: "fc4a7691-ea30-4bca-bc1b-0b195ef17a00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:57.095598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.078568    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:57.095598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.078606    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qsnh\" (UniqueName: \"kubernetes.io/projected/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-kube-api-access-7qsnh\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:57.095598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.078620    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:57.095598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.078632    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7691-ea30-4bca-bc1b-0b195ef17a00-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:57.119949 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.119567    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:57.408226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.408189    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh" event={"ID":"fc4a7691-ea30-4bca-bc1b-0b195ef17a00","Type":"ContainerDied","Data":"4b1cf336ae5255404cdfd507c8e38a9a565820ec3fdbbbe17c8371edc9932407"}
Oct 31 19:50:57.410094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.409035    5958 scope.go:122] "RemoveContainer" containerID="637dcbba0a35eceaeb3e4b57cfdf6b216f21f82335ede4f82189cb740e151090"
Oct 31 19:50:57.410094 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.408792    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"
Oct 31 19:50:57.413536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.413128    5958 generic.go:356] "Generic (PLEG): container finished" podID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerID="3b6bb0d11923e7a410828e0eca6e16fb62fc0dbdcacb247fbcc9d37ef86194bd" exitCode=0
Oct 31 19:50:57.413536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.413436    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" event={"ID":"5d5825f6-65d5-4cf0-aea6-bb08007987e8","Type":"ContainerDied","Data":"3b6bb0d11923e7a410828e0eca6e16fb62fc0dbdcacb247fbcc9d37ef86194bd"}
Oct 31 19:50:57.488135 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.488091    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"]
Oct 31 19:50:57.492457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.492425    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-s8rhh"]
Oct 31 19:50:57.637908 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.637424    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"]
Oct 31 19:50:57.643724 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:50:57.643690    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619b2c9c_f018_491e_b7e8_dae9abd7ccdc.slice/crio-959e5905e9ea406fb15a7382b76e8d9e3b0315283913f62634d7fc33823e620b WatchSource:0}: Error finding container 959e5905e9ea406fb15a7382b76e8d9e3b0315283913f62634d7fc33823e620b: Status 404 returned error can't find the container with id 959e5905e9ea406fb15a7382b76e8d9e3b0315283913f62634d7fc33823e620b
Oct 31 19:50:57.671723 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:50:57.671687    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37734fa_f33f_4d92_b6ce_0a81b16291d0.slice/crio-e6101e39756e87f0c6817f24dd346a111618f669db1cfcdeedf066985cd9aa05 WatchSource:0}: Error finding container e6101e39756e87f0c6817f24dd346a111618f669db1cfcdeedf066985cd9aa05: Status 404 returned error can't find the container with id e6101e39756e87f0c6817f24dd346a111618f669db1cfcdeedf066985cd9aa05
Oct 31 19:50:57.680660 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.680626    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"]
Oct 31 19:50:57.768766 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.768734    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:50:57.817622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.816592    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5\" (UniqueName: \"kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5\") pod \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") "
Oct 31 19:50:57.817622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.816658    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume\") pod \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\" (UID: \"5d5825f6-65d5-4cf0-aea6-bb08007987e8\") "
Oct 31 19:50:57.818491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.818434    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume" pod "5d5825f6-65d5-4cf0-aea6-bb08007987e8" (UID: "5d5825f6-65d5-4cf0-aea6-bb08007987e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:57.824639 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.824605    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5" pod "5d5825f6-65d5-4cf0-aea6-bb08007987e8" (UID: "5d5825f6-65d5-4cf0-aea6-bb08007987e8"). InnerVolumeSpecName "kube-api-access-sfln5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:57.920329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.920204    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfln5\" (UniqueName: \"kubernetes.io/projected/5d5825f6-65d5-4cf0-aea6-bb08007987e8-kube-api-access-sfln5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:57.920329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:57.920246    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5825f6-65d5-4cf0-aea6-bb08007987e8-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.046855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.046825    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:50:58.091022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.090995    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:50:58.122625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122587    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.122825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.122825 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122773    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.122952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122889    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.122952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122912    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.122952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122939    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.123109 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.122996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.123161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123143    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.123227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123170    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.123227 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123170    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.123341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123304    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs\") pod \"f128bd83-1027-45a5-b750-ce1920e34719\" (UID: \"f128bd83-1027-45a5-b750-ce1920e34719\") "
Oct 31 19:50:58.123341 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123309    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:58.123851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123742    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.124009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.123985    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.124270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.124207    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f128bd83-1027-45a5-b750-ce1920e34719-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.124270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.124234    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.124270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.124251    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.124270 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.124266    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.125318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.125279    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.125530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.125509    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:58.125926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.125906    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "kube-api-access-rdzrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:58.126005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.125940    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token" pod "f128bd83-1027-45a5-b750-ce1920e34719" (UID: "f128bd83-1027-45a5-b750-ce1920e34719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:58.225394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225356    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225412    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225460    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225477    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225519    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225610    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225723    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225791    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225817    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.225952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225872    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:58.225952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225890    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token\") pod \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\" (UID: \"aa4d71d6-13c0-439e-a7ba-3405108a2d8c\") "
Oct 31 19:50:58.225952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.225876    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.226274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226039    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.226274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226064    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.226669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226649    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226668    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226680    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzrb\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-kube-api-access-rdzrb\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226689    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f128bd83-1027-45a5-b750-ce1920e34719-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226699    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f128bd83-1027-45a5-b750-ce1920e34719-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226713    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226723    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226732    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226740    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.226850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.226749    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f128bd83-1027-45a5-b750-ce1920e34719-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.227978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.227944    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "kube-api-access-47dwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:58.228214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.228189    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:58.228387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.228302    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:58.228440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.228318    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo" pod "aa4d71d6-13c0-439e-a7ba-3405108a2d8c" (UID: "aa4d71d6-13c0-439e-a7ba-3405108a2d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:58.328514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.328480    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.328514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.328512    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.328514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.328522    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.328824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.328533    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47dwd\" (UniqueName: \"kubernetes.io/projected/aa4d71d6-13c0-439e-a7ba-3405108a2d8c-kube-api-access-47dwd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:58.424939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.424901    5958 generic.go:356] "Generic (PLEG): container finished" podID="f128bd83-1027-45a5-b750-ce1920e34719" containerID="61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696" exitCode=0
Oct 31 19:50:58.425384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.424957    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"
Oct 31 19:50:58.425384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.424982    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" event={"ID":"f128bd83-1027-45a5-b750-ce1920e34719","Type":"ContainerDied","Data":"61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696"}
Oct 31 19:50:58.425384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.425038    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb" event={"ID":"f128bd83-1027-45a5-b750-ce1920e34719","Type":"ContainerDied","Data":"acec092ef69f0de5f9ef913252a01c6e2950c87167cf93ac531c93f2d1aebcc0"}
Oct 31 19:50:58.425384 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.425069    5958 scope.go:122] "RemoveContainer" containerID="61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696"
Oct 31 19:50:58.426814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.426787    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" event={"ID":"b37734fa-f33f-4d92-b6ce-0a81b16291d0","Type":"ContainerStarted","Data":"abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9"}
Oct 31 19:50:58.426921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.426829    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" event={"ID":"b37734fa-f33f-4d92-b6ce-0a81b16291d0","Type":"ContainerStarted","Data":"e6101e39756e87f0c6817f24dd346a111618f669db1cfcdeedf066985cd9aa05"}
Oct 31 19:50:58.426921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.426887    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:50:58.428997 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.428968    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" event={"ID":"619b2c9c-f018-491e-b7e8-dae9abd7ccdc","Type":"ContainerStarted","Data":"333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4"}
Oct 31 19:50:58.429121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.429040    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" event={"ID":"619b2c9c-f018-491e-b7e8-dae9abd7ccdc","Type":"ContainerStarted","Data":"959e5905e9ea406fb15a7382b76e8d9e3b0315283913f62634d7fc33823e620b"}
Oct 31 19:50:58.429481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.429463    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:50:58.431912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.431895    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"
Oct 31 19:50:58.432020 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.431936    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz" event={"ID":"5d5825f6-65d5-4cf0-aea6-bb08007987e8","Type":"ContainerDied","Data":"d021ffabb40450a9d76ff176031468540ab60398eceebf68bcb72e225806e947"}
Oct 31 19:50:58.433842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.433460    5958 generic.go:356] "Generic (PLEG): container finished" podID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerID="7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d" exitCode=0
Oct 31 19:50:58.433842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.433501    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"
Oct 31 19:50:58.433842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.433592    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" event={"ID":"aa4d71d6-13c0-439e-a7ba-3405108a2d8c","Type":"ContainerDied","Data":"7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d"}
Oct 31 19:50:58.433842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.433628    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg" event={"ID":"aa4d71d6-13c0-439e-a7ba-3405108a2d8c","Type":"ContainerDied","Data":"85b4d09bc776afb74eee9973127bb9b7d95144eaf82df266d6586c8d886e14df"}
Oct 31 19:50:58.444041 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.443992    5958 scope.go:122] "RemoveContainer" containerID="61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696"
Oct 31 19:50:58.444352 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:58.444321    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696\": container with ID starting with 61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696 not found: ID does not exist" containerID="61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696"
Oct 31 19:50:58.444440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.444350    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696"} err="failed to get container status \"61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696\": rpc error: code = NotFound desc = could not find container \"61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696\": container with ID starting with 61293f67ed41d7a064d20564703e479d18acbb74a7a7a0b966a2dbbffd9d2696 not found: ID does not exist"
Oct 31 19:50:58.444440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.444372    5958 scope.go:122] "RemoveContainer" containerID="3b6bb0d11923e7a410828e0eca6e16fb62fc0dbdcacb247fbcc9d37ef86194bd"
Oct 31 19:50:58.457120 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.457096    5958 scope.go:122] "RemoveContainer" containerID="7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d"
Oct 31 19:50:58.469725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.469698    5958 scope.go:122] "RemoveContainer" containerID="7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d"
Oct 31 19:50:58.470072 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:50:58.470045    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d\": container with ID starting with 7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d not found: ID does not exist" containerID="7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d"
Oct 31 19:50:58.470178 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.470079    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d"} err="failed to get container status \"7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d\": rpc error: code = NotFound desc = could not find container \"7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d\": container with ID starting with 7c617001ba069e504b6b1c6311b945442895da2aa6193cb3e21fc8c99005570d not found: ID does not exist"
Oct 31 19:50:58.476029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.476000    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"]
Oct 31 19:50:58.480813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.480786    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-wt2tb"]
Oct 31 19:50:58.508832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.508762    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" podStartSLOduration=2.508749237 podStartE2EDuration="2.508749237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:58.508037719 +0000 UTC m=+1851.969412334" watchObservedRunningTime="2026-10-31 19:50:58.508749237 +0000 UTC m=+1851.970123851"
Oct 31 19:50:58.568287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.568232    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" podStartSLOduration=2.568218926 podStartE2EDuration="2.568218926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:58.567411112 +0000 UTC m=+1852.028785729" watchObservedRunningTime="2026-10-31 19:50:58.568218926 +0000 UTC m=+1852.029593542"
Oct 31 19:50:58.612131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.612090    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"]
Oct 31 19:50:58.621532 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.621486    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1fae7-fbbcfb98c-bvdgz"]
Oct 31 19:50:58.636931 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.636893    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"]
Oct 31 19:50:58.646469 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.646433    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-9vvhg"]
Oct 31 19:50:58.667177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.667144    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" path="/var/lib/kubelet/pods/5d5825f6-65d5-4cf0-aea6-bb08007987e8/volumes"
Oct 31 19:50:58.667846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.667824    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" path="/var/lib/kubelet/pods/aa4d71d6-13c0-439e-a7ba-3405108a2d8c/volumes"
Oct 31 19:50:58.668381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.668362    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f128bd83-1027-45a5-b750-ce1920e34719" path="/var/lib/kubelet/pods/f128bd83-1027-45a5-b750-ce1920e34719/volumes"
Oct 31 19:50:58.669250 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:50:58.669229    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" path="/var/lib/kubelet/pods/fc4a7691-ea30-4bca-bc1b-0b195ef17a00/volumes"
Oct 31 19:51:01.766197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:01.766144    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerName="payload-processing" containerID="cri-o://7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38" gracePeriod=25
Oct 31 19:51:01.776913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:01.776865    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerName="payload-pre-processing" containerID="cri-o://613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf" gracePeriod=25
Oct 31 19:51:02.243561 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:02.243514    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:03.576231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:03.576186    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"]
Oct 31 19:51:03.585671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:03.585634    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:04.556256 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:04.556217    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"]
Oct 31 19:51:08.577457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:08.577409    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerName="payload-pre-processing" containerID="cri-o://9581f5866e25ed7be70b11d47deebaf08f1b96dc2b921d4b5fc610818caad618" gracePeriod=25
Oct 31 19:51:09.450378 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:09.450348    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 19:51:09.630396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:09.630362    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"]
Oct 31 19:51:09.631175 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:09.631140    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj" podUID="f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" containerName="maas-api" containerID="cri-o://f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c" gracePeriod=30
Oct 31 19:51:10.215219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.215187    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:51:10.362950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.362860    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca\") pod \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") "
Oct 31 19:51:10.363121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.362988    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk\" (UniqueName: \"kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk\") pod \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") "
Oct 31 19:51:10.363121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.363104    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls\") pod \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") "
Oct 31 19:51:10.363277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.363259    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs\") pod \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\" (UID: \"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d\") "
Oct 31 19:51:10.363348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.363285    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca" pod "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" (UID: "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:10.364152 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.364120    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:10.365316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.365290    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls" pod "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" (UID: "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:10.365448 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.365422    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs" pod "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" (UID: "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:10.365503 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.365455    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk" pod "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" (UID: "f14218bf-3190-4d4e-a0a5-ce9c2f9b032d"). InnerVolumeSpecName "kube-api-access-lfnbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:10.460279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.460243    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 19:51:10.465990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.465954    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfnbk\" (UniqueName: \"kubernetes.io/projected/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-kube-api-access-lfnbk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:10.465990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.465985    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:10.465990 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.465995    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:10.675665 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.675626    5958 generic.go:356] "Generic (PLEG): container finished" podID="f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" containerID="f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c" exitCode=0
Oct 31 19:51:10.676142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.675678    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"
Oct 31 19:51:10.676142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.675710    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj" event={"ID":"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d","Type":"ContainerDied","Data":"f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c"}
Oct 31 19:51:10.676142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.675781    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj" event={"ID":"f14218bf-3190-4d4e-a0a5-ce9c2f9b032d","Type":"ContainerDied","Data":"6a04ba5c7d9b58116c71087457627fb764bc6d7637481b2e728d1af9088fb1cb"}
Oct 31 19:51:10.676142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.675822    5958 scope.go:122] "RemoveContainer" containerID="f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c"
Oct 31 19:51:10.687294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.687273    5958 scope.go:122] "RemoveContainer" containerID="f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c"
Oct 31 19:51:10.687603 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:10.687580    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c\": container with ID starting with f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c not found: ID does not exist" containerID="f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c"
Oct 31 19:51:10.687698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.687615    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c"} err="failed to get container status \"f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c\": rpc error: code = NotFound desc = could not find container \"f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c\": container with ID starting with f03efbe3d77949480edeee95c2364453a06975f5f05dc061247fbab43091291c not found: ID does not exist"
Oct 31 19:51:10.708725 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.708690    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"]
Oct 31 19:51:10.712792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:10.712769    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9efeae-79b7676d98-fwvfj"]
Oct 31 19:51:12.060814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:12.060779    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"]
Oct 31 19:51:12.243041 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:12.243002    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:12.671394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:12.671357    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" path="/var/lib/kubelet/pods/f14218bf-3190-4d4e-a0a5-ce9c2f9b032d/volumes"
Oct 31 19:51:13.274116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:13.274077    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:13.568123 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:13.568033    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:17.048866 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:17.048819    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:51:17.061301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.061245    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7" podUID="1aff6d5f-26da-423f-927d-37816b70747e" containerName="payload-processing" containerID="cri-o://cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f" gracePeriod=25
Oct 31 19:51:17.382723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.382692    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:51:17.507639 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.507606    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume\") pod \"1aff6d5f-26da-423f-927d-37816b70747e\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") "
Oct 31 19:51:17.507827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.507818    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln\" (UniqueName: \"kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln\") pod \"1aff6d5f-26da-423f-927d-37816b70747e\" (UID: \"1aff6d5f-26da-423f-927d-37816b70747e\") "
Oct 31 19:51:17.507971 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.507948    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume" pod "1aff6d5f-26da-423f-927d-37816b70747e" (UID: "1aff6d5f-26da-423f-927d-37816b70747e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:17.508438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.508420    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aff6d5f-26da-423f-927d-37816b70747e-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:17.510006 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.509980    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln" pod "1aff6d5f-26da-423f-927d-37816b70747e" (UID: "1aff6d5f-26da-423f-927d-37816b70747e"). InnerVolumeSpecName "kube-api-access-2m9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:17.609984 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.609902    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9ln\" (UniqueName: \"kubernetes.io/projected/1aff6d5f-26da-423f-927d-37816b70747e-kube-api-access-2m9ln\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:17.743464 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.743428    5958 generic.go:356] "Generic (PLEG): container finished" podID="1aff6d5f-26da-423f-927d-37816b70747e" containerID="cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f" exitCode=0
Oct 31 19:51:17.743714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.743508    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"
Oct 31 19:51:17.743793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.743746    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7" event={"ID":"1aff6d5f-26da-423f-927d-37816b70747e","Type":"ContainerDied","Data":"cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f"}
Oct 31 19:51:17.743855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.743804    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7" event={"ID":"1aff6d5f-26da-423f-927d-37816b70747e","Type":"ContainerDied","Data":"3489be8630888ebc9b75263289080db4deb9b017097555b708c25f82b06e7d74"}
Oct 31 19:51:17.743855 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.743835    5958 scope.go:122] "RemoveContainer" containerID="cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f"
Oct 31 19:51:17.768986 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.768855    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"]
Oct 31 19:51:17.769150 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.769107    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" podUID="53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" containerName="maas-api" containerID="cri-o://2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9" gracePeriod=30
Oct 31 19:51:17.770410 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.770381    5958 scope.go:122] "RemoveContainer" containerID="cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f"
Oct 31 19:51:17.770819 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:17.770794    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f\": container with ID starting with cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f not found: ID does not exist" containerID="cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f"
Oct 31 19:51:17.770924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.770829    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f"} err="failed to get container status \"cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f\": rpc error: code = NotFound desc = could not find container \"cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f\": container with ID starting with cee04a1d8c6f8028fc03509b48f3fa2d6dc2221e383f577ab23005515ddb2b5f not found: ID does not exist"
Oct 31 19:51:17.809976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.809945    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"]
Oct 31 19:51:17.824333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.824291    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d45f6c-59d8bb7487-tcsb7"]
Oct 31 19:51:17.930406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.930360    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"]
Oct 31 19:51:17.930747 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:17.930671    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy" containerID="cri-o://2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96" gracePeriod=30
Oct 31 19:51:18.227639 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.227516    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:51:18.243913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.243815    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs\") pod \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") "
Oct 31 19:51:18.244699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.244593    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca\") pod \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") "
Oct 31 19:51:18.246140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.244948    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\") pod \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") "
Oct 31 19:51:18.246140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.245024    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg\" (UniqueName: \"kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg\") pod \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\" (UID: \"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3\") "
Oct 31 19:51:18.262837 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.262801    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca" pod "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" (UID: "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:18.291996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.287137    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls" pod "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" (UID: "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:18.291996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.288746    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg" pod "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" (UID: "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3"). InnerVolumeSpecName "kube-api-access-cz6zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:18.291996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.288882    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs" pod "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" (UID: "53ea4bb4-066e-40d0-a94b-0acd3d0b91c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:18.352455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.352398    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:18.352692 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.352675    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:18.352814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.352799    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz6zg\" (UniqueName: \"kubernetes.io/projected/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-kube-api-access-cz6zg\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:18.352898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.352887    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:18.673111 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.673067    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aff6d5f-26da-423f-927d-37816b70747e" path="/var/lib/kubelet/pods/1aff6d5f-26da-423f-927d-37816b70747e/volumes"
Oct 31 19:51:18.754514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.754475    5958 generic.go:356] "Generic (PLEG): container finished" podID="53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" containerID="2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9" exitCode=0
Oct 31 19:51:18.754752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.754529    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"
Oct 31 19:51:18.754752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.754516    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" event={"ID":"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3","Type":"ContainerDied","Data":"2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9"}
Oct 31 19:51:18.754752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.754669    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26" event={"ID":"53ea4bb4-066e-40d0-a94b-0acd3d0b91c3","Type":"ContainerDied","Data":"35479bcec26305ae2661ddfe062e4da8928a91197c3b5e33a326de6284ed8105"}
Oct 31 19:51:18.754752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.754699    5958 scope.go:122] "RemoveContainer" containerID="2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9"
Oct 31 19:51:18.766335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.766308    5958 scope.go:122] "RemoveContainer" containerID="2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9"
Oct 31 19:51:18.766784 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:18.766762    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9\": container with ID starting with 2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9 not found: ID does not exist" containerID="2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9"
Oct 31 19:51:18.766887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.766791    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9"} err="failed to get container status \"2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9\": rpc error: code = NotFound desc = could not find container \"2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9\": container with ID starting with 2307c4d7af0a3c42b47e991f732f2a186b3aeaa7048db71904281238c643ddf9 not found: ID does not exist"
Oct 31 19:51:18.791105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.791076    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"]
Oct 31 19:51:18.795018 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:18.794994    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8c3f7-56754cdbdd-fff26"]
Oct 31 19:51:19.479535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.479499    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"]
Oct 31 19:51:19.481428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481363    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerName="payload-pre-processing"
Oct 31 19:51:19.481428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481394    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerName="payload-pre-processing"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481459    5958 cpu_manager.go:414] "containerMap: removing container" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481469    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481482    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481490    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481509    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" containerName="maas-api"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481516    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" containerName="maas-api"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481559    5958 cpu_manager.go:414] "containerMap: removing container" podUID="53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" containerName="maas-api"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481566    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" containerName="maas-api"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481602    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1aff6d5f-26da-423f-927d-37816b70747e" containerName="payload-processing"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481610    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aff6d5f-26da-423f-927d-37816b70747e" containerName="payload-processing"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481626    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy"
Oct 31 19:51:19.481684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481632    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481898    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14218bf-3190-4d4e-a0a5-ce9c2f9b032d" containerName="maas-api"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481920    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" containerName="maas-api"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481935    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aff6d5f-26da-423f-927d-37816b70747e" containerName="payload-processing"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481945    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f128bd83-1027-45a5-b750-ce1920e34719" containerName="istio-proxy"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481959    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5825f6-65d5-4cf0-aea6-bb08007987e8" containerName="payload-pre-processing"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481976    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4d71d6-13c0-439e-a7ba-3405108a2d8c" containerName="istio-proxy"
Oct 31 19:51:19.482100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.481986    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4a7691-ea30-4bca-bc1b-0b195ef17a00" containerName="istio-proxy"
Oct 31 19:51:19.488110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.488081    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.497218 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.496673    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c55cf1-serving-cert\""
Oct 31 19:51:19.501145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.500136    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"]
Oct 31 19:51:19.581051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.581016    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v42j\" (UniqueName: \"kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.581249 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.581096    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.581302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.581284    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.581522 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.581502    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.644578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.643893    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn" podUID="5394a82b-db7b-4626-957b-ea3b1b015613" containerName="main" containerID="cri-o://a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9" gracePeriod=45
Oct 31 19:51:19.686981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.684703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.686981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.684967    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.686981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.685045    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v42j\" (UniqueName: \"kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.686981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.685177    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.686981 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:19.686314    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-serving-cert: secret "maas-api-e2e-shared-a-w3-c55cf1-serving-cert" not found
Oct 31 19:51:19.686981 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:19.686396    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls podName:dc60ec47-7d8b-4643-8b18-5288ac867803 nodeName:}" failed. No retries permitted until 2026-10-31 19:51:20.186373879 +0000 UTC m=+1873.647748486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls") pod "maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" (UID: "dc60ec47-7d8b-4643-8b18-5288ac867803") : secret "maas-api-e2e-shared-a-w3-c55cf1-serving-cert" not found
Oct 31 19:51:19.688667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.688128    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.695326 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.695300    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.708296 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.708270    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v42j\" (UniqueName: \"kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:19.970525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.970493    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:51:19.992348 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992310    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home\") pod \"5394a82b-db7b-4626-957b-ea3b1b015613\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") "
Oct 31 19:51:19.992633 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992575    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir\") pod \"5394a82b-db7b-4626-957b-ea3b1b015613\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") "
Oct 31 19:51:19.992774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992643    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj\" (UniqueName: \"kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj\") pod \"5394a82b-db7b-4626-957b-ea3b1b015613\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") "
Oct 31 19:51:19.992774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992753    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home" pod "5394a82b-db7b-4626-957b-ea3b1b015613" (UID: "5394a82b-db7b-4626-957b-ea3b1b015613"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:19.992890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992780    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache\") pod \"5394a82b-db7b-4626-957b-ea3b1b015613\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") "
Oct 31 19:51:19.992890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992813    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs\" (UniqueName: \"kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs\") pod \"5394a82b-db7b-4626-957b-ea3b1b015613\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") "
Oct 31 19:51:19.992890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992883    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm\") pod \"5394a82b-db7b-4626-957b-ea3b1b015613\" (UID: \"5394a82b-db7b-4626-957b-ea3b1b015613\") "
Oct 31 19:51:19.993048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992923    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir" pod "5394a82b-db7b-4626-957b-ea3b1b015613" (UID: "5394a82b-db7b-4626-957b-ea3b1b015613"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:19.993048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.992988    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache" pod "5394a82b-db7b-4626-957b-ea3b1b015613" (UID: "5394a82b-db7b-4626-957b-ea3b1b015613"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:19.993576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.993524    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:19.993576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.993563    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:19.993682 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.993578    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:19.995100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.995078    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj" pod "5394a82b-db7b-4626-957b-ea3b1b015613" (UID: "5394a82b-db7b-4626-957b-ea3b1b015613"). InnerVolumeSpecName "kube-api-access-4hcxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:19.995482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.995463    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs" pod "5394a82b-db7b-4626-957b-ea3b1b015613" (UID: "5394a82b-db7b-4626-957b-ea3b1b015613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:19.995530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:19.995461    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm" pod "5394a82b-db7b-4626-957b-ea3b1b015613" (UID: "5394a82b-db7b-4626-957b-ea3b1b015613"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:20.095130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.095034    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5394a82b-db7b-4626-957b-ea3b1b015613-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:20.095130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.095070    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5394a82b-db7b-4626-957b-ea3b1b015613-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:20.095130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.095080    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hcxj\" (UniqueName: \"kubernetes.io/projected/5394a82b-db7b-4626-957b-ea3b1b015613-kube-api-access-4hcxj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:20.197032 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.196992    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:20.199509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.199481    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:20.411691 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.411661    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:20.666750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.666675    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ea4bb4-066e-40d0-a94b-0acd3d0b91c3" path="/var/lib/kubelet/pods/53ea4bb4-066e-40d0-a94b-0acd3d0b91c3/volumes"
Oct 31 19:51:20.717930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.717902    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"]
Oct 31 19:51:20.726270 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:51:20.726239    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc60ec47_7d8b_4643_8b18_5288ac867803.slice/crio-98755f95f2161ca043b6a329f121cf27d4171a073c5fc6c6a293159ec13f305d WatchSource:0}: Error finding container 98755f95f2161ca043b6a329f121cf27d4171a073c5fc6c6a293159ec13f305d: Status 404 returned error can't find the container with id 98755f95f2161ca043b6a329f121cf27d4171a073c5fc6c6a293159ec13f305d
Oct 31 19:51:20.774380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.774344    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" event={"ID":"dc60ec47-7d8b-4643-8b18-5288ac867803","Type":"ContainerStarted","Data":"98755f95f2161ca043b6a329f121cf27d4171a073c5fc6c6a293159ec13f305d"}
Oct 31 19:51:20.776364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.776336    5958 generic.go:356] "Generic (PLEG): container finished" podID="5394a82b-db7b-4626-957b-ea3b1b015613" containerID="a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9" exitCode=0
Oct 31 19:51:20.776499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.776384    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"
Oct 31 19:51:20.776499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.776413    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn" event={"ID":"5394a82b-db7b-4626-957b-ea3b1b015613","Type":"ContainerDied","Data":"a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9"}
Oct 31 19:51:20.776499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.776452    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn" event={"ID":"5394a82b-db7b-4626-957b-ea3b1b015613","Type":"ContainerDied","Data":"8ff776690fc3bfa694d3bd9c01f89286796b20509c862c8eae9edcbe892a8593"}
Oct 31 19:51:20.776499 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.776470    5958 scope.go:122] "RemoveContainer" containerID="a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9"
Oct 31 19:51:20.788154 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.788128    5958 scope.go:122] "RemoveContainer" containerID="a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9"
Oct 31 19:51:20.788468 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:20.788440    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9\": container with ID starting with a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9 not found: ID does not exist" containerID="a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9"
Oct 31 19:51:20.788620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.788473    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9"} err="failed to get container status \"a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9\": rpc error: code = NotFound desc = could not find container \"a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9\": container with ID starting with a5eec9611bc62b84bf1feeb4993dc66522bffa9e12079d397baa6069d8f31ce9 not found: ID does not exist"
Oct 31 19:51:20.847566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.845162    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"]
Oct 31 19:51:20.847566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:20.845227    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8dcfa8/test-model-8dcfa8-kserve-5dd474c887-rstpn"]
Oct 31 19:51:21.809350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:21.809309    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" event={"ID":"dc60ec47-7d8b-4643-8b18-5288ac867803","Type":"ContainerStarted","Data":"8273a3bcef2799ea1dbd689a414028e9e70ccb80ac7f335f46aa69bbe331884f"}
Oct 31 19:51:21.810130 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:21.810106    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:21.837191 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:21.837136    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" podStartSLOduration=2.434516245 podStartE2EDuration="2.837121059s" totalImagesPullingTime="402.604814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:21.832428232 +0000 UTC m=+1875.293802848" watchObservedRunningTime="2026-10-31 19:51:21.837121059 +0000 UTC m=+1875.298495674"
Oct 31 19:51:22.237335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.237300    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:51:22.241187 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.241165    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:51:22.346607 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.346497    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:51:22.393095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.393055    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv\" (UniqueName: \"kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv\") pod \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") "
Oct 31 19:51:22.393324 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.393285    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8\" (UniqueName: \"kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8\") pod \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") "
Oct 31 19:51:22.393375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.393329    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume\") pod \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\" (UID: \"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9\") "
Oct 31 19:51:22.393468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.393451    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume\") pod \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\" (UID: \"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb\") "
Oct 31 19:51:22.393874 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.393845    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume" pod "d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" (UID: "d18a1d4c-40bd-4c67-a9ee-0a2955323ea9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:22.393981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.393897    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume" pod "1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" (UID: "1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:22.394428 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.394403    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:22.394524 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.394433    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:22.395430 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.395409    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv" pod "d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" (UID: "d18a1d4c-40bd-4c67-a9ee-0a2955323ea9"). InnerVolumeSpecName "kube-api-access-stgcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:22.395539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.395519    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8" pod "1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" (UID: "1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb"). InnerVolumeSpecName "kube-api-access-254t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:22.496718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.496680    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254t8\" (UniqueName: \"kubernetes.io/projected/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb-kube-api-access-254t8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:22.496718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.496718    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stgcv\" (UniqueName: \"kubernetes.io/projected/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9-kube-api-access-stgcv\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:22.668589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.668538    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5394a82b-db7b-4626-957b-ea3b1b015613" path="/var/lib/kubelet/pods/5394a82b-db7b-4626-957b-ea3b1b015613/volumes"
Oct 31 19:51:22.823058 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.823024    5958 generic.go:356] "Generic (PLEG): container finished" podID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerID="613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf" exitCode=0
Oct 31 19:51:22.823519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.823073    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"
Oct 31 19:51:22.823519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.823109    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" event={"ID":"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb","Type":"ContainerDied","Data":"613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf"}
Oct 31 19:51:22.823519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.823173    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l" event={"ID":"1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb","Type":"ContainerDied","Data":"1aa764cf8a79804dac963a2e9b359a53bac0192e578ef5b467a3612fbe55e0c6"}
Oct 31 19:51:22.823519 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.823205    5958 scope.go:122] "RemoveContainer" containerID="613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf"
Oct 31 19:51:22.825027 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.825005    5958 generic.go:356] "Generic (PLEG): container finished" podID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerID="7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38" exitCode=0
Oct 31 19:51:22.825338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.825055    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"
Oct 31 19:51:22.825338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.825142    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" event={"ID":"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9","Type":"ContainerDied","Data":"7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38"}
Oct 31 19:51:22.825338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.825198    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc" event={"ID":"d18a1d4c-40bd-4c67-a9ee-0a2955323ea9","Type":"ContainerDied","Data":"26ffd2b4c62b92cda6915b32dacf2d5b042fec88e7d5796acab37f6d3152885b"}
Oct 31 19:51:22.835667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.835649    5958 scope.go:122] "RemoveContainer" containerID="613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf"
Oct 31 19:51:22.835962 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:22.835937    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf\": container with ID starting with 613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf not found: ID does not exist" containerID="613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf"
Oct 31 19:51:22.836031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.835969    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf"} err="failed to get container status \"613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf\": rpc error: code = NotFound desc = could not find container \"613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf\": container with ID starting with 613a4ab7e8d7a00343f3b90fc8fe5a98cde9a6cb5ce1e9f529609ee73328c5bf not found: ID does not exist"
Oct 31 19:51:22.836031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.835985    5958 scope.go:122] "RemoveContainer" containerID="7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38"
Oct 31 19:51:22.847876 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.847855    5958 scope.go:122] "RemoveContainer" containerID="7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38"
Oct 31 19:51:22.848216 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:22.848194    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38\": container with ID starting with 7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38 not found: ID does not exist" containerID="7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38"
Oct 31 19:51:22.848298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.848227    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38"} err="failed to get container status \"7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38\": rpc error: code = NotFound desc = could not find container \"7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38\": container with ID starting with 7e057f9421c8a85f0d88bdeed2029b4c10a82931e9ef4fbcef1c5fb0a3fc6a38 not found: ID does not exist"
Oct 31 19:51:22.849748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.849731    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"]
Oct 31 19:51:22.856166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.856139    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9efeae-6d89d89df7-7trdc"]
Oct 31 19:51:22.873016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.872982    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"]
Oct 31 19:51:22.896568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:22.884372    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9efeae-699f495dc5-bh64l"]
Oct 31 19:51:23.238254 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.238221    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:51:23.278288 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.278253    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:23.318996 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.318960    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319029    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319092    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319211 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319157    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319445 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319232    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319445 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319379    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.319445 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319428    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319490    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319539    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.319656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.319600    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo\") pod \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\" (UID: \"cd2b36c8-ea98-4c3f-80b7-75cd130861b4\") "
Oct 31 19:51:23.320085 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.320045    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.320313 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.320291    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.320394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.320341    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.320394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.320376    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.321836 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.321782    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.322156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.322121    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:23.325585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.325526    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "kube-api-access-kkmrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:23.325733 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.325657    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:23.326143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.326045    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:23.326143 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.326122    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy" pod "cd2b36c8-ea98-4c3f-80b7-75cd130861b4" (UID: "cd2b36c8-ea98-4c3f-80b7-75cd130861b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.421774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421740    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.421774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421769    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.421774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421779    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.422009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421788    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.422009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421798    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.422009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421807    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmrk\" (UniqueName: \"kubernetes.io/projected/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-kube-api-access-kkmrk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.422009 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.421816    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd2b36c8-ea98-4c3f-80b7-75cd130861b4-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.835966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.835932    5958 generic.go:356] "Generic (PLEG): container finished" podID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerID="2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96" exitCode=0
Oct 31 19:51:23.836434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.835975    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"
Oct 31 19:51:23.836434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.836015    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" event={"ID":"cd2b36c8-ea98-4c3f-80b7-75cd130861b4","Type":"ContainerDied","Data":"2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96"}
Oct 31 19:51:23.836434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.836064    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772" event={"ID":"cd2b36c8-ea98-4c3f-80b7-75cd130861b4","Type":"ContainerDied","Data":"e8a0c19b3e21c1441e1a5df64a92c8e61664c9f6646c25290acc367eb9e65865"}
Oct 31 19:51:23.836434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.836087    5958 scope.go:122] "RemoveContainer" containerID="2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96"
Oct 31 19:51:23.848789 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.848763    5958 scope.go:122] "RemoveContainer" containerID="2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96"
Oct 31 19:51:23.849114 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:23.849095    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96\": container with ID starting with 2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96 not found: ID does not exist" containerID="2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96"
Oct 31 19:51:23.849170 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.849122    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96"} err="failed to get container status \"2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96\": rpc error: code = NotFound desc = could not find container \"2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96\": container with ID starting with 2ed621ff6f384d8c5eb40cf37ce5e3a74007442346d8f3593517144ed1087a96 not found: ID does not exist"
Oct 31 19:51:23.920647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.920612    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"]
Oct 31 19:51:23.928383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:23.928351    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4g772"]
Oct 31 19:51:24.667028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:24.666991    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" path="/var/lib/kubelet/pods/1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb/volumes"
Oct 31 19:51:24.667732 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:24.667714    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" path="/var/lib/kubelet/pods/cd2b36c8-ea98-4c3f-80b7-75cd130861b4/volumes"
Oct 31 19:51:24.668342 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:24.668325    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" path="/var/lib/kubelet/pods/d18a1d4c-40bd-4c67-a9ee-0a2955323ea9/volumes"
Oct 31 19:51:29.042568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.042408    5958 generic.go:356] "Generic (PLEG): container finished" podID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerID="9581f5866e25ed7be70b11d47deebaf08f1b96dc2b921d4b5fc610818caad618" exitCode=0
Oct 31 19:51:29.055566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.052225    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 19:51:29.055566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.052285    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" event={"ID":"ddad558e-416e-460f-8d10-1c7b965efb7e","Type":"ContainerDied","Data":"9581f5866e25ed7be70b11d47deebaf08f1b96dc2b921d4b5fc610818caad618"}
Oct 31 19:51:29.410849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.409071    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:51:29.534234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.534179    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7\" (UniqueName: \"kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7\") pod \"ddad558e-416e-460f-8d10-1c7b965efb7e\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") "
Oct 31 19:51:29.534451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.534326    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume\") pod \"ddad558e-416e-460f-8d10-1c7b965efb7e\" (UID: \"ddad558e-416e-460f-8d10-1c7b965efb7e\") "
Oct 31 19:51:29.539022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.538812    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume" pod "ddad558e-416e-460f-8d10-1c7b965efb7e" (UID: "ddad558e-416e-460f-8d10-1c7b965efb7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:29.539022 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.538919    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7" pod "ddad558e-416e-460f-8d10-1c7b965efb7e" (UID: "ddad558e-416e-460f-8d10-1c7b965efb7e"). InnerVolumeSpecName "kube-api-access-bpbj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:29.637192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.637155    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbj7\" (UniqueName: \"kubernetes.io/projected/ddad558e-416e-460f-8d10-1c7b965efb7e-kube-api-access-bpbj7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:29.637192 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:29.637195    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddad558e-416e-460f-8d10-1c7b965efb7e-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:30.056088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:30.056004    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z" event={"ID":"ddad558e-416e-460f-8d10-1c7b965efb7e","Type":"ContainerDied","Data":"4a37f1d31d816fb04a8ecaebab1e14a1c09eac2d26579a8d83d978bcc86f925f"}
Oct 31 19:51:30.056088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:30.056035    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"
Oct 31 19:51:30.056088 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:30.056067    5958 scope.go:122] "RemoveContainer" containerID="9581f5866e25ed7be70b11d47deebaf08f1b96dc2b921d4b5fc610818caad618"
Oct 31 19:51:30.093701 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:30.093663    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"]
Oct 31 19:51:30.099680 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:30.099646    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8c3f7-d97564b6c-z2z2z"]
Oct 31 19:51:30.667578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:30.667523    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" path="/var/lib/kubelet/pods/ddad558e-416e-460f-8d10-1c7b965efb7e/volumes"
Oct 31 19:51:41.407699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.407661    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"]
Oct 31 19:51:41.408843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408824    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerName="payload-pre-processing"
Oct 31 19:51:41.408924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408845    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerName="payload-pre-processing"
Oct 31 19:51:41.408924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408872    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerName="payload-pre-processing"
Oct 31 19:51:41.408924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408880    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerName="payload-pre-processing"
Oct 31 19:51:41.409025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408927    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5394a82b-db7b-4626-957b-ea3b1b015613" containerName="main"
Oct 31 19:51:41.409025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408934    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5394a82b-db7b-4626-957b-ea3b1b015613" containerName="main"
Oct 31 19:51:41.409025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408958    5958 cpu_manager.go:414] "containerMap: removing container" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy"
Oct 31 19:51:41.409025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408965    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy"
Oct 31 19:51:41.409025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408980    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerName="payload-processing"
Oct 31 19:51:41.409025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.408987    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerName="payload-processing"
Oct 31 19:51:41.409212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.409182    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5394a82b-db7b-4626-957b-ea3b1b015613" containerName="main"
Oct 31 19:51:41.409212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.409197    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2b36c8-ea98-4c3f-80b7-75cd130861b4" containerName="istio-proxy"
Oct 31 19:51:41.409212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.409208    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18a1d4c-40bd-4c67-a9ee-0a2955323ea9" containerName="payload-processing"
Oct 31 19:51:41.409345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.409228    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddad558e-416e-460f-8d10-1c7b965efb7e" containerName="payload-pre-processing"
Oct 31 19:51:41.409345 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.409241    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfb7ff5-e74c-4b2a-b1b0-b3d954287adb" containerName="payload-pre-processing"
Oct 31 19:51:41.426572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.421930    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.552580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.533761    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.552580 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.533820    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4wl8\" (UniqueName: \"kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8\") pod \"payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.602120 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.602083    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-e618e1-dockercfg-mt9h5\""
Oct 31 19:51:41.606566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.602897    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-e618e1\""
Oct 31 19:51:41.638164 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.637078    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.638688 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.638514    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4wl8\" (UniqueName: \"kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8\") pod \"payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.640639 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.640609    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.679053 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.678974    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"]
Oct 31 19:51:41.684609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.684525    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4wl8\" (UniqueName: \"kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8\") pod \"payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:41.687241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.686350    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.687453 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.686254    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"]
Oct 31 19:51:41.691153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.691131    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"]
Oct 31 19:51:41.740591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.740512    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.740773 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.740651    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4d98\" (UniqueName: \"kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98\") pod \"payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.843213 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.843180    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.843397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.843241    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4d98\" (UniqueName: \"kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98\") pod \"payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.844371 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.844301    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.857015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.856988    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4d98\" (UniqueName: \"kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98\") pod \"payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:41.940779 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:41.930875    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:42.022045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:42.021886    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:42.366618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:42.365420    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"]
Oct 31 19:51:42.378875 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:42.378843    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"]
Oct 31 19:51:42.383712 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:51:42.383676    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66eba36_4f37_41dd_86e0_35987e97506d.slice/crio-d221c7e3c77ff8bea32f361f866232cbecdaa5191c34c500f3befee1d7d4141f WatchSource:0}: Error finding container d221c7e3c77ff8bea32f361f866232cbecdaa5191c34c500f3befee1d7d4141f: Status 404 returned error can't find the container with id d221c7e3c77ff8bea32f361f866232cbecdaa5191c34c500f3befee1d7d4141f
Oct 31 19:51:43.351590 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.351539    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" event={"ID":"3d461c42-cd97-42d7-9098-b4911126ce01","Type":"ContainerStarted","Data":"d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951"}
Oct 31 19:51:43.352076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.351617    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" event={"ID":"3d461c42-cd97-42d7-9098-b4911126ce01","Type":"ContainerStarted","Data":"2aefb9d6120340658f8cda82f107256ffd2a94983fcba701567a86b8c350bddc"}
Oct 31 19:51:43.352076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.351647    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:43.353452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.353422    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" event={"ID":"b66eba36-4f37-41dd-86e0-35987e97506d","Type":"ContainerStarted","Data":"a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae"}
Oct 31 19:51:43.353597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.353473    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" event={"ID":"b66eba36-4f37-41dd-86e0-35987e97506d","Type":"ContainerStarted","Data":"d221c7e3c77ff8bea32f361f866232cbecdaa5191c34c500f3befee1d7d4141f"}
Oct 31 19:51:43.353597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.353502    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:43.378476 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.378417    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" podStartSLOduration=2.378403207 podStartE2EDuration="2.378403207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:43.374463769 +0000 UTC m=+1896.835838384" watchObservedRunningTime="2026-10-31 19:51:43.378403207 +0000 UTC m=+1896.839777836"
Oct 31 19:51:43.412468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:43.412408    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" podStartSLOduration=2.412395531 podStartE2EDuration="2.412395531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:43.411935476 +0000 UTC m=+1896.873310090" watchObservedRunningTime="2026-10-31 19:51:43.412395531 +0000 UTC m=+1896.873770147"
Oct 31 19:51:47.052089 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:51:47.052002    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:51:52.993898 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:52.993860    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"]
Oct 31 19:51:53.000420 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.000393    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.012851 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.012822    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae4ce2\""
Oct 31 19:51:53.015888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.015855    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"]
Oct 31 19:51:53.017327 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.017300    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae4ce2-dockercfg-hpd4q\""
Oct 31 19:51:53.021113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.021003    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qbr\" (UniqueName: \"kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr\") pod \"payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.021561 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.021300    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.023976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.023949    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"]
Oct 31 19:51:53.030822 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.030800    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.065848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.065816    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"]
Oct 31 19:51:53.197251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.197213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.230785 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.230732    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.231242 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.231208    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qbr\" (UniqueName: \"kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr\") pod \"payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.231538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.231516    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sms97\" (UniqueName: \"kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97\") pod \"payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.232745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.211819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.329119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.329027    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qbr\" (UniqueName: \"kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr\") pod \"payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.333203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.333159    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.335117 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.335089    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sms97\" (UniqueName: \"kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97\") pod \"payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.335672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.334329    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.350922 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.350830    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sms97\" (UniqueName: \"kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97\") pod \"payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.377708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.377670    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:53.621355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.621310    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:53.728807 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:51:53.728773    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba25a2f1_c066_4891_a93c_0e20f9e74418.slice/crio-270ac188d60d314bdd791523aa73feed1acde369edf24d7d860b34e1713b64bf WatchSource:0}: Error finding container 270ac188d60d314bdd791523aa73feed1acde369edf24d7d860b34e1713b64bf: Status 404 returned error can't find the container with id 270ac188d60d314bdd791523aa73feed1acde369edf24d7d860b34e1713b64bf
Oct 31 19:51:53.729163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.729139    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"]
Oct 31 19:51:53.983060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:53.983026    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"]
Oct 31 19:51:53.983429 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:51:53.983407    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69cfbd1_5c8b_473e_8244_48b487c848a2.slice/crio-73376dcc9f9f3ddee6bee9240b9fc2b3e8e5f8517db08b5381e53a1e49c62adb WatchSource:0}: Error finding container 73376dcc9f9f3ddee6bee9240b9fc2b3e8e5f8517db08b5381e53a1e49c62adb: Status 404 returned error can't find the container with id 73376dcc9f9f3ddee6bee9240b9fc2b3e8e5f8517db08b5381e53a1e49c62adb
Oct 31 19:51:54.364582 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.364476    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 19:51:54.364955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.364760    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 19:51:54.491193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.491158    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs" event={"ID":"b69cfbd1-5c8b-473e-8244-48b487c848a2","Type":"ContainerStarted","Data":"a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2"}
Oct 31 19:51:54.491379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.491222    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs" event={"ID":"b69cfbd1-5c8b-473e-8244-48b487c848a2","Type":"ContainerStarted","Data":"73376dcc9f9f3ddee6bee9240b9fc2b3e8e5f8517db08b5381e53a1e49c62adb"}
Oct 31 19:51:54.491379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.491251    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:51:54.492933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.492905    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" event={"ID":"ba25a2f1-c066-4891-a93c-0e20f9e74418","Type":"ContainerStarted","Data":"ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682"}
Oct 31 19:51:54.493057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.492951    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" event={"ID":"ba25a2f1-c066-4891-a93c-0e20f9e74418","Type":"ContainerStarted","Data":"270ac188d60d314bdd791523aa73feed1acde369edf24d7d860b34e1713b64bf"}
Oct 31 19:51:54.493057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.492975    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:51:54.514652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.514600    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs" podStartSLOduration=2.5145862450000003 podStartE2EDuration="2.514586245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:54.511824518 +0000 UTC m=+1907.973199135" watchObservedRunningTime="2026-10-31 19:51:54.514586245 +0000 UTC m=+1907.975960860"
Oct 31 19:51:54.535739 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:51:54.535676    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" podStartSLOduration=2.5356605439999997 podStartE2EDuration="2.535660544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:54.532624265 +0000 UTC m=+1907.993998884" watchObservedRunningTime="2026-10-31 19:51:54.535660544 +0000 UTC m=+1907.997035158"
Oct 31 19:52:00.586800 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.586696    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"]
Oct 31 19:52:00.606089 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.601327    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.606635 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.606467    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"]
Oct 31 19:52:00.608316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.608287    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-e618e1-serving-cert\""
Oct 31 19:52:00.740929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.740273    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.740929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.740509    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswkd\" (UniqueName: \"kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.740929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.740564    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.740929 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.740631    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.842917 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.842831    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.843350 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.843324    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswkd\" (UniqueName: \"kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.843457 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.843387    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.843518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.843505    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.843715 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.843641    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.846245 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.846212    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.846449 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.846434    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.859661 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.859637    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswkd\" (UniqueName: \"kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd\") pod \"maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:00.921197 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:00.921161    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:01.246119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:01.246087    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"]
Oct 31 19:52:01.247374 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:52:01.247346    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f1bebb_6d87_417c_88b7_4e41031f6a83.slice/crio-8ec4f060da2a986ad185404ab23545c69abe09c2daa32da7cb8cb2da7b52111f WatchSource:0}: Error finding container 8ec4f060da2a986ad185404ab23545c69abe09c2daa32da7cb8cb2da7b52111f: Status 404 returned error can't find the container with id 8ec4f060da2a986ad185404ab23545c69abe09c2daa32da7cb8cb2da7b52111f
Oct 31 19:52:01.249273 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:01.249250    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:52:01.570533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:01.570452    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478" event={"ID":"f5f1bebb-6d87-417c-88b7-4e41031f6a83","Type":"ContainerStarted","Data":"8ec4f060da2a986ad185404ab23545c69abe09c2daa32da7cb8cb2da7b52111f"}
Oct 31 19:52:02.581873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:02.581821    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478" event={"ID":"f5f1bebb-6d87-417c-88b7-4e41031f6a83","Type":"ContainerStarted","Data":"24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e"}
Oct 31 19:52:02.582372 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:02.581906    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:05.505229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:05.505196    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 19:52:05.505700 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:05.505602    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 19:52:05.538334 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:05.538272    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478" podStartSLOduration=5.07465598 podStartE2EDuration="5.538257142s" totalImagesPullingTime="463.601162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:02.62272043 +0000 UTC m=+1916.084095047" watchObservedRunningTime="2026-10-31 19:52:05.538257142 +0000 UTC m=+1918.999631756"
Oct 31 19:52:05.911050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:05.911012    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"]
Oct 31 19:52:08.596339 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:08.596305    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 19:52:11.181316 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.181281    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"]
Oct 31 19:52:11.191460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.191426    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.198209 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.198175    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ae4ce2-serving-cert\""
Oct 31 19:52:11.200650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.200613    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"]
Oct 31 19:52:11.336575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.329037    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.336575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.329332    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c89b\" (UniqueName: \"kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.336575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.329380    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.336575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.329419    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.430939 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.430900    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.431125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.431106    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c89b\" (UniqueName: \"kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.431177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.431154    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.431215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.431193    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.431422 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:52:11.431366    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-serving-cert: secret "maas-api-e2e-shared-b-w3-ae4ce2-serving-cert" not found
Oct 31 19:52:11.431509 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:52:11.431455    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls podName:420f78c4-3d9b-43b3-8115-f00fa70f956e nodeName:}" failed. No retries permitted until 2026-10-31 19:52:11.931435477 +0000 UTC m=+1925.392810089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" (UID: "420f78c4-3d9b-43b3-8115-f00fa70f956e") : secret "maas-api-e2e-shared-b-w3-ae4ce2-serving-cert" not found
Oct 31 19:52:11.431724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.431701    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.433574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.433556    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.449846 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.449820    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c89b\" (UniqueName: \"kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.950068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.950021    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:11.958565 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:11.958038    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:12.120907 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:12.119878    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:12.452340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:12.452305    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"]
Oct 31 19:52:12.457056 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:52:12.457024    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420f78c4_3d9b_43b3_8115_f00fa70f956e.slice/crio-6cbba8ef52052d91295d69f62f6bdfbe4ff4201d3b053fee65266c180035d2f7 WatchSource:0}: Error finding container 6cbba8ef52052d91295d69f62f6bdfbe4ff4201d3b053fee65266c180035d2f7: Status 404 returned error can't find the container with id 6cbba8ef52052d91295d69f62f6bdfbe4ff4201d3b053fee65266c180035d2f7
Oct 31 19:52:12.713475 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:12.713368    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" event={"ID":"420f78c4-3d9b-43b3-8115-f00fa70f956e","Type":"ContainerStarted","Data":"6cbba8ef52052d91295d69f62f6bdfbe4ff4201d3b053fee65266c180035d2f7"}
Oct 31 19:52:13.724062 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:13.724019    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" event={"ID":"420f78c4-3d9b-43b3-8115-f00fa70f956e","Type":"ContainerStarted","Data":"4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f"}
Oct 31 19:52:13.724483 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:13.724118    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:13.750761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:13.750686    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" podStartSLOduration=2.341384308 podStartE2EDuration="2.750669967s" totalImagesPullingTime="409.285659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:13.748669192 +0000 UTC m=+1927.210043809" watchObservedRunningTime="2026-10-31 19:52:13.750669967 +0000 UTC m=+1927.212044582"
Oct 31 19:52:17.046898 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:52:17.046857    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:52:19.738676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:19.738646    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 19:52:20.939622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:20.939585    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq" podUID="9f21647a-426d-44b2-a060-e2c0e4a0ce53" containerName="main" containerID="cri-o://ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1" gracePeriod=45
Oct 31 19:52:21.330393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.330365    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:52:21.491161 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491121    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir\") pod \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") "
Oct 31 19:52:21.491357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491205    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm\") pod \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") "
Oct 31 19:52:21.491357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491253    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs\") pod \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") "
Oct 31 19:52:21.491357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491277    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8\" (UniqueName: \"kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8\") pod \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") "
Oct 31 19:52:21.491614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491445    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir" pod "9f21647a-426d-44b2-a060-e2c0e4a0ce53" (UID: "9f21647a-426d-44b2-a060-e2c0e4a0ce53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.491614 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491483    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache\") pod \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") "
Oct 31 19:52:21.491755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491641    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home\") pod \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\" (UID: \"9f21647a-426d-44b2-a060-e2c0e4a0ce53\") "
Oct 31 19:52:21.491809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491771    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache" pod "9f21647a-426d-44b2-a060-e2c0e4a0ce53" (UID: "9f21647a-426d-44b2-a060-e2c0e4a0ce53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.492301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.491907    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home" pod "9f21647a-426d-44b2-a060-e2c0e4a0ce53" (UID: "9f21647a-426d-44b2-a060-e2c0e4a0ce53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.492301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.492254    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.492301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.492269    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.492301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.492279    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.494285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.493849    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm" pod "9f21647a-426d-44b2-a060-e2c0e4a0ce53" (UID: "9f21647a-426d-44b2-a060-e2c0e4a0ce53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.494285 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.493960    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs" pod "9f21647a-426d-44b2-a060-e2c0e4a0ce53" (UID: "9f21647a-426d-44b2-a060-e2c0e4a0ce53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:52:21.494862 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.494832    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8" pod "9f21647a-426d-44b2-a060-e2c0e4a0ce53" (UID: "9f21647a-426d-44b2-a060-e2c0e4a0ce53"). InnerVolumeSpecName "kube-api-access-2wzm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:21.596158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.596027    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f21647a-426d-44b2-a060-e2c0e4a0ce53-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.596158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.596065    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f21647a-426d-44b2-a060-e2c0e4a0ce53-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.596158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.596081    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzm8\" (UniqueName: \"kubernetes.io/projected/9f21647a-426d-44b2-a060-e2c0e4a0ce53-kube-api-access-2wzm8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.837402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.836990    5958 generic.go:356] "Generic (PLEG): container finished" podID="9f21647a-426d-44b2-a060-e2c0e4a0ce53" containerID="ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1" exitCode=0
Oct 31 19:52:21.837402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.837091    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq" event={"ID":"9f21647a-426d-44b2-a060-e2c0e4a0ce53","Type":"ContainerDied","Data":"ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1"}
Oct 31 19:52:21.837402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.837131    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq" event={"ID":"9f21647a-426d-44b2-a060-e2c0e4a0ce53","Type":"ContainerDied","Data":"10b613be3783531deaeceaf3a99a61c625a14dc4e978ee57e8e40b917fa2420d"}
Oct 31 19:52:21.837402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.837153    5958 scope.go:122] "RemoveContainer" containerID="ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1"
Oct 31 19:52:21.837402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.837335    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"
Oct 31 19:52:21.900240 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.895140    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"]
Oct 31 19:52:21.902463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.902431    5958 scope.go:122] "RemoveContainer" containerID="ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1"
Oct 31 19:52:21.913814 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:52:21.912438    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1\": container with ID starting with ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1 not found: ID does not exist" containerID="ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1"
Oct 31 19:52:21.913814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.912475    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1"} err="failed to get container status \"ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1\": rpc error: code = NotFound desc = could not find container \"ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1\": container with ID starting with ffa58104cc11b6e0de9488834d725a28c654dfb63808d9c63ac16203fea079e1 not found: ID does not exist"
Oct 31 19:52:21.913814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:21.913725    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-785d26/test-model-785d26-kserve-5b9cfbff79-l7zhq"]
Oct 31 19:52:22.671839 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:22.671797    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f21647a-426d-44b2-a060-e2c0e4a0ce53" path="/var/lib/kubelet/pods/9f21647a-426d-44b2-a060-e2c0e4a0ce53/volumes"
Oct 31 19:52:23.453460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:23.453422    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:52:47.046855 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:52:47.046819    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:52:52.416460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.416417    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"]
Oct 31 19:52:52.417349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.417333    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9f21647a-426d-44b2-a060-e2c0e4a0ce53" containerName="main"
Oct 31 19:52:52.417349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.417350    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f21647a-426d-44b2-a060-e2c0e4a0ce53" containerName="main"
Oct 31 19:52:52.417514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.417505    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f21647a-426d-44b2-a060-e2c0e4a0ce53" containerName="main"
Oct 31 19:52:52.423000 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.422978    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.437395 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.437358    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"]
Oct 31 19:52:52.503943 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.503888    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxfv\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504039    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504206 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504104    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504289 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504259    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504289 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504276    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504355 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504311    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504354    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504413    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.504478 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.504459    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.625665    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.625737    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626006    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626140    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626212    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626282    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626435    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.626571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.626565    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxfv\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.627532    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.627804    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.628454    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.628729    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.629440    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.633896    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.650568 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.637170    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.666622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.666357    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.686606 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.674866    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxfv\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:52.743853 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:52.743828    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:53.069439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.069311    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"]
Oct 31 19:52:53.074399 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:52:53.074350    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f3d6c3_bf3b_4570_a4ff_31c27131ade5.slice/crio-17ce8122568e48870de5fd09454844e98aa4c322b9c1811981254046c0fcf6be WatchSource:0}: Error finding container 17ce8122568e48870de5fd09454844e98aa4c322b9c1811981254046c0fcf6be: Status 404 returned error can't find the container with id 17ce8122568e48870de5fd09454844e98aa4c322b9c1811981254046c0fcf6be
Oct 31 19:52:53.078583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.078245    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:52:53.078583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.078326    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:52:53.078583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.078383    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:52:53.213445 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.213407    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" event={"ID":"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5","Type":"ContainerStarted","Data":"3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c"}
Oct 31 19:52:53.213827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.213805    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" event={"ID":"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5","Type":"ContainerStarted","Data":"17ce8122568e48870de5fd09454844e98aa4c322b9c1811981254046c0fcf6be"}
Oct 31 19:52:53.250072 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.250019    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podStartSLOduration=1.250005284 podStartE2EDuration="1.250005284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:53.245286529 +0000 UTC m=+1966.706661155" watchObservedRunningTime="2026-10-31 19:52:53.250005284 +0000 UTC m=+1966.711379899"
Oct 31 19:52:53.744215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:53.744181    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:54.747101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:54.747068    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:52:54.747578 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:54.747123    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:52:55.744442 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:55.744407    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:52:55.744664 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:55.744460    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:52:56.744866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:56.744833    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:52:56.745290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:56.744895    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:52:57.745093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:57.745063    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:52:57.745589 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:57.745114    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:52:58.745213 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:58.745177    5958 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:52:58.745710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:58.745227    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 19:52:59.782869 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:59.782837    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:59.783439 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:59.783411    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:52:59.784381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:52:59.784308    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:53:05.371535 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.371486    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"]
Oct 31 19:53:05.381592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.381236    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.389451 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.389418    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"]
Oct 31 19:53:05.409656 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.409630    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae4ce2\"/\"openshift-service-ca.crt\""
Oct 31 19:53:05.409821 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.409719    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae4ce2\"/\"default-dockercfg-8frtr\""
Oct 31 19:53:05.411882 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.411855    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae4ce2\"/\"kube-root-ca.crt\""
Oct 31 19:53:05.411882 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.411855    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae4ce2\"/\"auth-test-model-ae4ce2-kserve-self-signed-certs\""
Oct 31 19:53:05.445912 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.445876    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.446093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.446014    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.446093 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.446050    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.446189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.446171    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.446287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.446273    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576mq\" (UniqueName: \"kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.446353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.446338    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.547826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.547790    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548030 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.547888    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548030 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.547918    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548030 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.547949    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.548051    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576mq\" (UniqueName: \"kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548214 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.548141    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.548260    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.548674    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.548827 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.548796    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.550858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.550811    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.551520 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.551474    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.582087 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.582033    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576mq\" (UniqueName: \"kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq\") pod \"auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") " pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:05.700872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:05.700775    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:06.063073 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:53:06.063028    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a7c51b_ab5b_4557_baa7_29a70955e3cb.slice/crio-bc50d0c000eddde4129afebe3e6dfdafc24d16b71477610576faf1654c5b46c9 WatchSource:0}: Error finding container bc50d0c000eddde4129afebe3e6dfdafc24d16b71477610576faf1654c5b46c9: Status 404 returned error can't find the container with id bc50d0c000eddde4129afebe3e6dfdafc24d16b71477610576faf1654c5b46c9
Oct 31 19:53:06.064470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:06.064447    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"]
Oct 31 19:53:06.359290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:06.359201    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" event={"ID":"b5a7c51b-ab5b-4557-baa7-29a70955e3cb","Type":"ContainerStarted","Data":"d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0"}
Oct 31 19:53:06.359290 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:06.359244    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" event={"ID":"b5a7c51b-ab5b-4557-baa7-29a70955e3cb","Type":"ContainerStarted","Data":"bc50d0c000eddde4129afebe3e6dfdafc24d16b71477610576faf1654c5b46c9"}
Oct 31 19:53:06.425567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:06.424603    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" podStartSLOduration=1.424579389 podStartE2EDuration="1.424579389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:06.396995239 +0000 UTC m=+1979.858369856" watchObservedRunningTime="2026-10-31 19:53:06.424579389 +0000 UTC m=+1979.885954005"
Oct 31 19:53:06.702200 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:06.702171    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:09.456257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:09.456177    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"]
Oct 31 19:53:14.456816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.456776    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754" podUID="2ed33b55-7692-4bbc-bcd0-82522ef33dab" containerName="payload-processing" containerID="cri-o://cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302" gracePeriod=25
Oct 31 19:53:14.763843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.763811    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:53:14.864767 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.864729    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume\") pod \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") "
Oct 31 19:53:14.864965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.864832    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn\" (UniqueName: \"kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn\") pod \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\" (UID: \"2ed33b55-7692-4bbc-bcd0-82522ef33dab\") "
Oct 31 19:53:14.865275 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.865243    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume" pod "2ed33b55-7692-4bbc-bcd0-82522ef33dab" (UID: "2ed33b55-7692-4bbc-bcd0-82522ef33dab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:14.865460 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.865439    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ed33b55-7692-4bbc-bcd0-82522ef33dab-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:14.867824 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.867796    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn" pod "2ed33b55-7692-4bbc-bcd0-82522ef33dab" (UID: "2ed33b55-7692-4bbc-bcd0-82522ef33dab"). InnerVolumeSpecName "kube-api-access-nhbbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:14.967259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:14.967176    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhbbn\" (UniqueName: \"kubernetes.io/projected/2ed33b55-7692-4bbc-bcd0-82522ef33dab-kube-api-access-nhbbn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:15.436174 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.436136    5958 generic.go:356] "Generic (PLEG): container finished" podID="2ed33b55-7692-4bbc-bcd0-82522ef33dab" containerID="cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302" exitCode=0
Oct 31 19:53:15.436383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.436185    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"
Oct 31 19:53:15.436383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.436221    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754" event={"ID":"2ed33b55-7692-4bbc-bcd0-82522ef33dab","Type":"ContainerDied","Data":"cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302"}
Oct 31 19:53:15.436383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.436273    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754" event={"ID":"2ed33b55-7692-4bbc-bcd0-82522ef33dab","Type":"ContainerDied","Data":"4d2f8aeed2f7837540ed46146d6f145ad61f980de46c98e614bef4ca269bd152"}
Oct 31 19:53:15.436383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.436298    5958 scope.go:122] "RemoveContainer" containerID="cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302"
Oct 31 19:53:15.449142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.449117    5958 scope.go:122] "RemoveContainer" containerID="cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302"
Oct 31 19:53:15.449434 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:53:15.449413    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302\": container with ID starting with cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302 not found: ID does not exist" containerID="cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302"
Oct 31 19:53:15.449506 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.449447    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302"} err="failed to get container status \"cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302\": rpc error: code = NotFound desc = could not find container \"cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302\": container with ID starting with cd7c4608d42eaaf13a524a60f61bbe6f91a3378c1eb5f9242304dd7b16eb0302 not found: ID does not exist"
Oct 31 19:53:15.465973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.465941    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"]
Oct 31 19:53:15.471338 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.471315    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d8c398-546ff884f-pr754"]
Oct 31 19:53:15.702301 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.702218    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:15.715297 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:15.715269    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:16.457772 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:16.457644    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 19:53:16.666921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:16.666888    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed33b55-7692-4bbc-bcd0-82522ef33dab" path="/var/lib/kubelet/pods/2ed33b55-7692-4bbc-bcd0-82522ef33dab/volumes"
Oct 31 19:53:17.048291 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:53:17.048235    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:53:23.639927 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:23.639892    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:53:36.984902 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:36.984861    5958 generic.go:356] "Generic (PLEG): container finished" podID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerID="e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1" exitCode=0
Oct 31 19:53:36.985352 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:36.984930    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" event={"ID":"e1fa028a-9dc6-4935-9456-1dc22bfb2578","Type":"ContainerDied","Data":"e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"}
Oct 31 19:53:36.985433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:36.985416    5958 scope.go:122] "RemoveContainer" containerID="e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"
Oct 31 19:53:36.987224 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:36.987195    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:53:36.987294 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:36.987265    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:53:36.987334 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:36.987296    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:53:37.994806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:37.994770    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" event={"ID":"e1fa028a-9dc6-4935-9456-1dc22bfb2578","Type":"ContainerStarted","Data":"e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c"}
Oct 31 19:53:38.663611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:38.662718    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 19:53:38.663611 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:38.662777    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 19:53:38.682975 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:38.682944    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:53:38.830835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:38.830798    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:53:39.635572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:39.631751    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 19:53:39.635572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:39.631813    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 19:53:40.629563 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:40.628691    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 19:53:40.629563 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:40.628753    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 19:53:41.629587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:41.629108    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 19:53:41.629587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:41.629192    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 19:53:42.629066 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:42.629033    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 19:53:42.629279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:42.629109    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 19:53:43.627151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:43.627115    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 31 19:53:43.627593 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:43.627203    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 31 19:53:44.633601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:44.633532    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:53:44.634892 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:53:44.634868    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 19:53:47.048773 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:53:47.048728    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:54:17.059095 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:54:17.059033    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:54:47.048379 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:54:47.048286    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:54:58.837806 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:54:58.837767    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:55:17.049522 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:55:17.049484    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:55:34.097485 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.097435    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"]
Oct 31 19:55:34.098394 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.098377    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2ed33b55-7692-4bbc-bcd0-82522ef33dab" containerName="payload-processing"
Oct 31 19:55:34.098455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.098405    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed33b55-7692-4bbc-bcd0-82522ef33dab" containerName="payload-processing"
Oct 31 19:55:34.098612 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.098602    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed33b55-7692-4bbc-bcd0-82522ef33dab" containerName="payload-processing"
Oct 31 19:55:34.102693 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.102678    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.121061 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.121034    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"]
Oct 31 19:55:34.167068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167028    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167191    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167279 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167220    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167494 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167467    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167594    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167656    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167712    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167739    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zns55\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.167934 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.167841    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.192880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.192843    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ae2e60-openshift-default-dockercfg-ts6pq\""
Oct 31 19:55:34.269386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269342    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269409    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269466    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269495    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zns55\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269631 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269602    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269647    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269781    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269813    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.269888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269818    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.270095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.269997    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.270177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.270145    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.270177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.270172    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.270412 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.270290    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.270412 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.270313    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.272257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.272236    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.272470 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.272452    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.283758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.283736    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.283981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.283958    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zns55\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.420525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.420495    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:34.804151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.804117    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"]
Oct 31 19:55:34.809408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.809352    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:55:34.809577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.809509    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:55:34.809657 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:34.809623    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:55:35.227887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:35.227849    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" event={"ID":"0592a59d-4a48-432c-acef-7dab11bf3809","Type":"ContainerStarted","Data":"6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d"}
Oct 31 19:55:35.228437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:35.227914    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" event={"ID":"0592a59d-4a48-432c-acef-7dab11bf3809","Type":"ContainerStarted","Data":"6abbe792c6a11b753697270d7b084121854094fc2d98a71a898339baef3564ff"}
Oct 31 19:55:35.420741 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:35.420705    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:35.422318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:35.422287    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:35.422482 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:35.422354    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:36.431570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:36.430933    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:36.431570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:36.431011    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:37.422575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:37.422130    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:37.422575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:37.422247    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:38.428744 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:38.428700    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:38.429268 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:38.428819    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:39.425757 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:39.425709    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:39.426084 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:39.426040    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:40.432716 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:40.432677    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:40.433226 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:40.433122    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:41.421407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:41.421371    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:55:41.421621 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:41.421426    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Oct 31 19:55:42.426098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:42.426062    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:42.426835 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:42.426810    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:42.427515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:42.427496    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 19:55:42.471948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:42.471889    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podStartSLOduration=8.471875618 podStartE2EDuration="8.471875618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:35.300921597 +0000 UTC m=+2128.762296216" watchObservedRunningTime="2026-10-31 19:55:42.471875618 +0000 UTC m=+2135.933250233"
Oct 31 19:55:47.058224 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:55:47.058185    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:55:49.147371 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.147337    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"]
Oct 31 19:55:49.153585 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.153563    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.215344 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.215293    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"]
Oct 31 19:55:49.280443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280408    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.280669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280462    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.280669 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280581    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.280780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280710    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gn8p\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.280780 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280761    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.280901 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280880    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.280974 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.280959    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.281037 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.281009    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.281095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.281083    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382170 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382133    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382228    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gn8p\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382273    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382333    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382378    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382421    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382714 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382473    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382730    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382831 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382773    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.382969 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382937    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.383034 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.382971    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.383113 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.383088    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.383203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.383182    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.383282 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.383264    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.384708 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.384687    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.385096 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.385077    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.396787 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.396761    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.398751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.398698    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gn8p\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.497177 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.497135    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:49.821923 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:55:49.821888    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a1856b_cd18_4bfb_9a25_68175c3858e6.slice/crio-bcbb279d950f731f6de4b612a4c2423d932ee025eaa3e721da7dded850cdadc8 WatchSource:0}: Error finding container bcbb279d950f731f6de4b612a4c2423d932ee025eaa3e721da7dded850cdadc8: Status 404 returned error can't find the container with id bcbb279d950f731f6de4b612a4c2423d932ee025eaa3e721da7dded850cdadc8
Oct 31 19:55:49.822866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.822758    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"]
Oct 31 19:55:49.825238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.825205    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:55:49.825346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.825273    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:55:49.825346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:49.825300    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:55:50.422474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:50.422434    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" event={"ID":"83a1856b-cd18-4bfb-9a25-68175c3858e6","Type":"ContainerStarted","Data":"ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60"}
Oct 31 19:55:50.422973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:50.422504    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" event={"ID":"83a1856b-cd18-4bfb-9a25-68175c3858e6","Type":"ContainerStarted","Data":"bcbb279d950f731f6de4b612a4c2423d932ee025eaa3e721da7dded850cdadc8"}
Oct 31 19:55:50.455166 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:50.455113    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podStartSLOduration=1.455099731 podStartE2EDuration="1.455099731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:50.452856758 +0000 UTC m=+2143.914231374" watchObservedRunningTime="2026-10-31 19:55:50.455099731 +0000 UTC m=+2143.916474346"
Oct 31 19:55:50.498156 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:50.498122    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:50.499807 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:50.499781    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:50.499942 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:50.499830    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:51.497832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:51.497801    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:51.498244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:51.497851    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:52.498010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:52.497966    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:52.498435 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:52.498031    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:53.498149 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:53.498114    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:53.498566 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:53.498168    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:54.498246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:54.498208    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:54.498686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:54.498264    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:55.497733 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:55.497696    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:55.497915 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:55.497754    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:56.558577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:56.543492    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:55:56.559745 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:56.559705    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused"
Oct 31 19:55:57.636647 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:57.636606    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:57.637659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:57.637605    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:57.640730 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:57.640699    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 19:55:59.108829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:59.108791    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"]
Oct 31 19:55:59.152233 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:55:59.152164    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"]
Oct 31 19:56:04.110063 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.110019    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerName="payload-processing" containerID="cri-o://4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4" gracePeriod=25
Oct 31 19:56:04.153408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.153337    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerName="payload-pre-processing" containerID="cri-o://1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c" gracePeriod=25
Oct 31 19:56:04.181151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.181120    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"]
Oct 31 19:56:04.187576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.187518    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.217404 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.217354    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"]
Oct 31 19:56:04.288652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.288572    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.288652 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.288658    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289067 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.288912    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289067 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.288980    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9clf\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289196 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.289126    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289272 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.289213    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289559 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.289505    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289724 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.289694    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.289888 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.289863    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391354    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391417    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391443    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391594    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391629    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9clf\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391784    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391887    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.391906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391902    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.392121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391985    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.392121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.391993    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.392121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.392086    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.392375 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.392352    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.392447 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.392354    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.393920 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.393894    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.394351 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.394333    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.406051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.406001    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9clf\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.406051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.406033    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.511144 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.511112    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:04.812474 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.812433    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"]
Oct 31 19:56:04.813668 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:56:04.813639    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdec16c6_aa60_40ff_a953_e3c291e05e24.slice/crio-91074c33675e770cab13f66305db4b3f242816f971c1e61db41e13d1e94fad21 WatchSource:0}: Error finding container 91074c33675e770cab13f66305db4b3f242816f971c1e61db41e13d1e94fad21: Status 404 returned error can't find the container with id 91074c33675e770cab13f66305db4b3f242816f971c1e61db41e13d1e94fad21
Oct 31 19:56:04.816893 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.816856    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:56:04.817029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.816951    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:56:04.817029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:04.816998    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:56:05.786575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:05.768096    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" event={"ID":"cdec16c6-aa60-40ff-a953-e3c291e05e24","Type":"ContainerStarted","Data":"09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e"}
Oct 31 19:56:05.786575 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:05.768220    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" event={"ID":"cdec16c6-aa60-40ff-a953-e3c291e05e24","Type":"ContainerStarted","Data":"91074c33675e770cab13f66305db4b3f242816f971c1e61db41e13d1e94fad21"}
Oct 31 19:56:05.851527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:05.851448    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podStartSLOduration=1.851423637 podStartE2EDuration="1.851423637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:05.842841681 +0000 UTC m=+2159.304216297" watchObservedRunningTime="2026-10-31 19:56:05.851423637 +0000 UTC m=+2159.312798254"
Oct 31 19:56:06.531609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:06.520613    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:06.531609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:06.521166    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:56:06.531609 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:06.521210    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:56:07.518577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:07.515606    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:56:07.518577 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:07.515669    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:56:08.534045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:08.524167    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:56:08.534045 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:08.524228    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:56:09.521570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:09.519709    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:56:09.521570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:09.519796    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:56:10.527569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:10.518577    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:56:10.527569 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:10.518650    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:56:11.512234 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:11.512192    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Oct 31 19:56:11.512429 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:11.512263    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Oct 31 19:56:12.516438 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:12.516403    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:12.516926 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:12.516907    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:12.517592 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:12.517569    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 19:56:13.180624 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.180590    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"]
Oct 31 19:56:13.180849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.180810    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c" podUID="efc76981-8b90-47e1-bd2a-c82e9ec017e6" containerName="maas-api" containerID="cri-o://fa51681f8ed25fdeac80e506de771f1bb705be0fede7ea0fe586eefcd0494548" gracePeriod=30
Oct 31 19:56:13.876274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.876239    5958 generic.go:356] "Generic (PLEG): container finished" podID="efc76981-8b90-47e1-bd2a-c82e9ec017e6" containerID="fa51681f8ed25fdeac80e506de771f1bb705be0fede7ea0fe586eefcd0494548" exitCode=0
Oct 31 19:56:13.876712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.876337    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c" event={"ID":"efc76981-8b90-47e1-bd2a-c82e9ec017e6","Type":"ContainerDied","Data":"fa51681f8ed25fdeac80e506de771f1bb705be0fede7ea0fe586eefcd0494548"}
Oct 31 19:56:13.876712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.876424    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c" event={"ID":"efc76981-8b90-47e1-bd2a-c82e9ec017e6","Type":"ContainerDied","Data":"809dc078e519fe03e9a5c1b8cc4d333c84d2e104f2e364720b3853167ed68350"}
Oct 31 19:56:13.876712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.876466    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="809dc078e519fe03e9a5c1b8cc4d333c84d2e104f2e364720b3853167ed68350"
Oct 31 19:56:13.890356 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.890325    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:56:13.931961 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.931922    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs\") pod \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") "
Oct 31 19:56:13.932153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.932008    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls\") pod \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") "
Oct 31 19:56:13.932153 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.932113    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca\") pod \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") "
Oct 31 19:56:13.932255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.932241    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp\" (UniqueName: \"kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp\") pod \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\" (UID: \"efc76981-8b90-47e1-bd2a-c82e9ec017e6\") "
Oct 31 19:56:13.935689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.935651    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca" pod "efc76981-8b90-47e1-bd2a-c82e9ec017e6" (UID: "efc76981-8b90-47e1-bd2a-c82e9ec017e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:13.936374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.936350    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp" pod "efc76981-8b90-47e1-bd2a-c82e9ec017e6" (UID: "efc76981-8b90-47e1-bd2a-c82e9ec017e6"). InnerVolumeSpecName "kube-api-access-nllhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:13.936489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.936430    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls" pod "efc76981-8b90-47e1-bd2a-c82e9ec017e6" (UID: "efc76981-8b90-47e1-bd2a-c82e9ec017e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:13.936924 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:13.936896    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs" pod "efc76981-8b90-47e1-bd2a-c82e9ec017e6" (UID: "efc76981-8b90-47e1-bd2a-c82e9ec017e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:14.026116 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.026069    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:56:14.026309 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.026223    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:56:14.034463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.034430    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:14.034463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.034459    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc76981-8b90-47e1-bd2a-c82e9ec017e6-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:14.034463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.034471    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc76981-8b90-47e1-bd2a-c82e9ec017e6-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:14.034756 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.034481    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nllhp\" (UniqueName: \"kubernetes.io/projected/efc76981-8b90-47e1-bd2a-c82e9ec017e6-kube-api-access-nllhp\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:14.884965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.884934    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"
Oct 31 19:56:14.910303 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.910268    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"]
Oct 31 19:56:14.916601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:14.916574    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-754e0a-d5874d588-8294c"]
Oct 31 19:56:16.669155 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:16.669067    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc76981-8b90-47e1-bd2a-c82e9ec017e6" path="/var/lib/kubelet/pods/efc76981-8b90-47e1-bd2a-c82e9ec017e6/volumes"
Oct 31 19:56:17.048076 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:17.047988    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:56:21.959025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.958980    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"]
Oct 31 19:56:21.960246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.960217    5958 cpu_manager.go:414] "containerMap: removing container" podUID="efc76981-8b90-47e1-bd2a-c82e9ec017e6" containerName="maas-api"
Oct 31 19:56:21.960246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.960244    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc76981-8b90-47e1-bd2a-c82e9ec017e6" containerName="maas-api"
Oct 31 19:56:21.960461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.960449    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc76981-8b90-47e1-bd2a-c82e9ec017e6" containerName="maas-api"
Oct 31 19:56:21.966443 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.966418    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:21.971858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.971831    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e618e1-openshift-default-dockercfg-h496q\""
Oct 31 19:56:21.980259 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:21.980226    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"]
Oct 31 19:56:22.106241 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106201    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106247    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106433 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106391    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106527 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106512    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106622 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106606    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106709 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106677    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106833    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5pk\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106883    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.106966 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.106935    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.208991 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.208952    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209081    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209212 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209196    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5pk\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209234    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209268    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209353 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209333    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209357    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209446    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209503    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209593    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209712 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209685    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209729    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209776    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.209930 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.209829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.211842 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.211818    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.211985 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.211968    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.253940 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.253907    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5pk\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.254287 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.254270    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.285362 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.285329    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:22.584778 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:56:22.584738    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9193f73f_a5b0_4033_ac97_94c990e4019c.slice/crio-3457b0e43b716ee924cce9a7547d8ccfc39bfdb4ae1249c2818ef128bbffdcc2 WatchSource:0}: Error finding container 3457b0e43b716ee924cce9a7547d8ccfc39bfdb4ae1249c2818ef128bbffdcc2: Status 404 returned error can't find the container with id 3457b0e43b716ee924cce9a7547d8ccfc39bfdb4ae1249c2818ef128bbffdcc2
Oct 31 19:56:22.585095 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.585073    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"]
Oct 31 19:56:22.587683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.587650    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:56:22.587788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.587720    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:56:22.587788 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.587751    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:56:22.953236 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.953197    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" event={"ID":"9193f73f-a5b0-4033-ac97-94c990e4019c","Type":"ContainerStarted","Data":"fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d"}
Oct 31 19:56:22.953422 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:22.953264    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" event={"ID":"9193f73f-a5b0-4033-ac97-94c990e4019c","Type":"ContainerStarted","Data":"3457b0e43b716ee924cce9a7547d8ccfc39bfdb4ae1249c2818ef128bbffdcc2"}
Oct 31 19:56:23.000768 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:23.000706    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podStartSLOduration=2.000689911 podStartE2EDuration="2.000689911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:22.99622915 +0000 UTC m=+2176.457603815" watchObservedRunningTime="2026-10-31 19:56:23.000689911 +0000 UTC m=+2176.462064525"
Oct 31 19:56:23.286178 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:23.286094    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:23.288125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:23.288096    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:23.288257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:23.288163    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:24.031026 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.030976    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:56:24.031658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.031468    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:56:24.269179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.269141    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"]
Oct 31 19:56:24.269418 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.269390    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy" containerID="cri-o://abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb" gracePeriod=30
Oct 31 19:56:24.274031 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.273984    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"]
Oct 31 19:56:24.274598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.274226    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy" containerID="cri-o://3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c" gracePeriod=30
Oct 31 19:56:24.287251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.287146    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:24.289392 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.288594    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:24.839834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.839702    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:56:24.881959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.880879    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume\") pod \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") "
Oct 31 19:56:24.881959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.880996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq\" (UniqueName: \"kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq\") pod \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\" (UID: \"f0381a0e-ac17-456e-81fc-d283cb5e62ae\") "
Oct 31 19:56:24.882640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.882614    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume" pod "f0381a0e-ac17-456e-81fc-d283cb5e62ae" (UID: "f0381a0e-ac17-456e-81fc-d283cb5e62ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:24.954440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.954408    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq" pod "f0381a0e-ac17-456e-81fc-d283cb5e62ae" (UID: "f0381a0e-ac17-456e-81fc-d283cb5e62ae"). InnerVolumeSpecName "kube-api-access-vc5vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:24.983437 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.983402    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0381a0e-ac17-456e-81fc-d283cb5e62ae-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:24.983634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.983451    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc5vq\" (UniqueName: \"kubernetes.io/projected/f0381a0e-ac17-456e-81fc-d283cb5e62ae-kube-api-access-vc5vq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:24.987906 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.987873    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:56:24.994415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.994381    5958 generic.go:356] "Generic (PLEG): container finished" podID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerID="4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4" exitCode=0
Oct 31 19:56:24.994710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.994683    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"
Oct 31 19:56:24.994913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.994685    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" event={"ID":"f0381a0e-ac17-456e-81fc-d283cb5e62ae","Type":"ContainerDied","Data":"4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4"}
Oct 31 19:56:24.995016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.994956    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8" event={"ID":"f0381a0e-ac17-456e-81fc-d283cb5e62ae","Type":"ContainerDied","Data":"1e49c52d7e815714f70a507162c7cfad828bdad00173a68737839f9355daea71"}
Oct 31 19:56:24.995016 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.994986    5958 scope.go:122] "RemoveContainer" containerID="4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4"
Oct 31 19:56:24.996648 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.996625    5958 generic.go:356] "Generic (PLEG): container finished" podID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerID="1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c" exitCode=0
Oct 31 19:56:24.996781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.996671    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"
Oct 31 19:56:24.996781 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.996753    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" event={"ID":"87973e9e-4ad8-48d1-8714-975d9bad5403","Type":"ContainerDied","Data":"1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c"}
Oct 31 19:56:24.996911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:24.996789    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb" event={"ID":"87973e9e-4ad8-48d1-8714-975d9bad5403","Type":"ContainerDied","Data":"38002fcee8c6d2ea8258971fef2816645abbfe2c601dc3cd5bce2cbf92fdbca8"}
Oct 31 19:56:25.045699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.045670    5958 scope.go:122] "RemoveContainer" containerID="4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4"
Oct 31 19:56:25.046139 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:25.046101    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4\": container with ID starting with 4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4 not found: ID does not exist" containerID="4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4"
Oct 31 19:56:25.046216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.046136    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4"} err="failed to get container status \"4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4\": rpc error: code = NotFound desc = could not find container \"4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4\": container with ID starting with 4e7f31462820d9ccb4be6c9cf150f6180d010149b604bd0c1c2ecdc68de4ceb4 not found: ID does not exist"
Oct 31 19:56:25.046216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.046159    5958 scope.go:122] "RemoveContainer" containerID="1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c"
Oct 31 19:56:25.059406 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.059351    5958 scope.go:122] "RemoveContainer" containerID="1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c"
Oct 31 19:56:25.059726 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:25.059696    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c\": container with ID starting with 1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c not found: ID does not exist" containerID="1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c"
Oct 31 19:56:25.059832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.059727    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c"} err="failed to get container status \"1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c\": rpc error: code = NotFound desc = could not find container \"1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c\": container with ID starting with 1490421dc82b39a3cc129f07ee33d608698a72ee05299bff51ebc72479f4d05c not found: ID does not exist"
Oct 31 19:56:25.074619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.073597    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"]
Oct 31 19:56:25.080761 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.080719    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-754e0a-85f75ff745-b8jz8"]
Oct 31 19:56:25.084704 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.084675    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume\") pod \"87973e9e-4ad8-48d1-8714-975d9bad5403\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") "
Oct 31 19:56:25.084872 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.084725    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr\" (UniqueName: \"kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr\") pod \"87973e9e-4ad8-48d1-8714-975d9bad5403\" (UID: \"87973e9e-4ad8-48d1-8714-975d9bad5403\") "
Oct 31 19:56:25.085073 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.085050    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume" pod "87973e9e-4ad8-48d1-8714-975d9bad5403" (UID: "87973e9e-4ad8-48d1-8714-975d9bad5403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:25.085587 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.085571    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87973e9e-4ad8-48d1-8714-975d9bad5403-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:25.087142 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.087120    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr" pod "87973e9e-4ad8-48d1-8714-975d9bad5403" (UID: "87973e9e-4ad8-48d1-8714-975d9bad5403"). InnerVolumeSpecName "kube-api-access-wxnvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:25.187075 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.186987    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxnvr\" (UniqueName: \"kubernetes.io/projected/87973e9e-4ad8-48d1-8714-975d9bad5403-kube-api-access-wxnvr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:25.291261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.287983    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:25.291261 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.288052    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:25.399557 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.399419    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"]
Oct 31 19:56:25.418750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:25.418715    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-754e0a-7858857447-zl6nb"]
Oct 31 19:56:26.289619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:26.288053    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:26.289619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:26.288134    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:26.667229 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:26.667195    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" path="/var/lib/kubelet/pods/87973e9e-4ad8-48d1-8714-975d9bad5403/volumes"
Oct 31 19:56:26.667694 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:26.667679    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" path="/var/lib/kubelet/pods/f0381a0e-ac17-456e-81fc-d283cb5e62ae/volumes"
Oct 31 19:56:27.293634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:27.290893    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:27.293634 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:27.290952    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:28.285695 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:28.285653    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:28.285891 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:28.285714    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:29.286048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.286006    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:56:29.286509 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.286107    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 31 19:56:29.662328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.662115    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:56:29.722193 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722157    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722200    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722284    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722366 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722343    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722425    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722481    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722567    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722595    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.722812 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.722786    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj\") pod \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\" (UID: \"a5f38528-4ce5-4c6f-8fb4-833bba442ff8\") "
Oct 31 19:56:29.723101 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.723073    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.723495 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.723471    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.724189 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.724162    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.724291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.724192    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.730686 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.730656    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "kube-api-access-9zxqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.731865 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.731835    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.732239 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.732213    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:29.732320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.732281    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.735950 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.735924    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.737671 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.737647    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.739092 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.739066    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo" pod "a5f38528-4ce5-4c6f-8fb4-833bba442ff8" (UID: "a5f38528-4ce5-4c6f-8fb4-833bba442ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:29.744620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.743846    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:56:29.825947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.825730    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.825947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.825799    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.825947 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.825899    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.826264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.825968    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.826381 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.826353    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.826481 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.826464    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.826556 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.826523    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.826620 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.826611    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.826677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.826665    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket\") pod \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\" (UID: \"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5\") "
Oct 31 19:56:29.827364 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.827305    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.827496    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.827763    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828065    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828135    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxqj\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-kube-api-access-9zxqj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828167    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828214    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828231    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828246    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828260    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828276    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f38528-4ce5-4c6f-8fb4-833bba442ff8-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828292    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828307    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828322    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.829518 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.828593    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.830302 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.830087    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "kube-api-access-9pxfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.831455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.831310    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.832539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.832501    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.834050 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.834021    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo" pod "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" (UID: "e5f3d6c3-bf3b-4570-a4ff-31c27131ade5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:29.931405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.931347    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.931405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.931392    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pxfv\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-kube-api-access-9pxfv\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.931405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.931408    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.931911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.931420    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.931911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.931434    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.931911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:29.931447    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:30.122857 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.122818    5958 generic.go:356] "Generic (PLEG): container finished" podID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerID="abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb" exitCode=0
Oct 31 19:56:30.123060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.122892    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"
Oct 31 19:56:30.123060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.122908    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" event={"ID":"a5f38528-4ce5-4c6f-8fb4-833bba442ff8","Type":"ContainerDied","Data":"abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb"}
Oct 31 19:56:30.123060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.123020    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r" event={"ID":"a5f38528-4ce5-4c6f-8fb4-833bba442ff8","Type":"ContainerDied","Data":"551ebec6bf89abce35738e235a8f402bd0e51961a817831b7b7883729364ac00"}
Oct 31 19:56:30.123060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.123053    5958 scope.go:122] "RemoveContainer" containerID="abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb"
Oct 31 19:56:30.124625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.124590    5958 generic.go:356] "Generic (PLEG): container finished" podID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerID="3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c" exitCode=0
Oct 31 19:56:30.124775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.124657    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" event={"ID":"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5","Type":"ContainerDied","Data":"3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c"}
Oct 31 19:56:30.124775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.124693    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl" event={"ID":"e5f3d6c3-bf3b-4570-a4ff-31c27131ade5","Type":"ContainerDied","Data":"17ce8122568e48870de5fd09454844e98aa4c322b9c1811981254046c0fcf6be"}
Oct 31 19:56:30.124775 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.124701    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"
Oct 31 19:56:30.139572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.139441    5958 scope.go:122] "RemoveContainer" containerID="abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb"
Oct 31 19:56:30.139946 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:30.139912    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb\": container with ID starting with abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb not found: ID does not exist" containerID="abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb"
Oct 31 19:56:30.140112 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.140081    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb"} err="failed to get container status \"abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb\": rpc error: code = NotFound desc = could not find container \"abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb\": container with ID starting with abe45e0ffd77af233cf228bb03754605fb216df5db15d0a4d6bebe5693e1efcb not found: ID does not exist"
Oct 31 19:56:30.140216 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.140115    5958 scope.go:122] "RemoveContainer" containerID="3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c"
Oct 31 19:56:30.154752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.154723    5958 scope.go:122] "RemoveContainer" containerID="3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c"
Oct 31 19:56:30.155150 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:30.155126    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c\": container with ID starting with 3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c not found: ID does not exist" containerID="3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c"
Oct 31 19:56:30.155231 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.155159    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c"} err="failed to get container status \"3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c\": rpc error: code = NotFound desc = could not find container \"3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c\": container with ID starting with 3bf0c246536eb3a6b07e1258ff0d3ef186c12a218cf958871ece83f8a5c5a79c not found: ID does not exist"
Oct 31 19:56:30.155480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.155462    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"]
Oct 31 19:56:30.161795 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.161768    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-b594r"]
Oct 31 19:56:30.176440 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.176399    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"]
Oct 31 19:56:30.182667 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.182633    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-sljnl"]
Oct 31 19:56:30.291389 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.291352    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:30.291854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.291808    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:30.292645 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.292624    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 19:56:30.668591 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.668532    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" path="/var/lib/kubelet/pods/a5f38528-4ce5-4c6f-8fb4-833bba442ff8/volumes"
Oct 31 19:56:30.669158 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:30.669142    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" path="/var/lib/kubelet/pods/e5f3d6c3-bf3b-4570-a4ff-31c27131ade5/volumes"
Oct 31 19:56:47.059463 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:47.059421    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:56:51.610076 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:51.610044    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"]
Oct 31 19:56:51.610676 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:51.610273    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy" containerID="cri-o://ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511" gracePeriod=30
Oct 31 19:56:52.810181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:52.810147    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:56:52.857195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:52.857160    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"]
Oct 31 19:56:52.857397 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:52.857374    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy" containerID="cri-o://9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0" gracePeriod=30
Oct 31 19:56:56.938911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:56.938880    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:56:57.099257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099225    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099257 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099266    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099349    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099516 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099489    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099658 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099530    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099727 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099701    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099792 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099769    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.099848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099789    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099815    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.099848 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099816    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:57.100021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099906    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket\") pod \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\" (UID: \"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4\") "
Oct 31 19:56:57.100021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.099960    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.100163 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100117    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.100264 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100231    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.100642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100621    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.100642 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100640    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.100816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100650    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.100816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100658    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.100816 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.100666    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.103005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.102979    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:57.103005 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.103000    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:57.103155 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.102984    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "kube-api-access-4jx45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:57.103155 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.102986    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy" pod "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" (UID: "5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.202386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.202303    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.202386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.202336    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jx45\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-kube-api-access-4jx45\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.202386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.202352    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.202386 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.202363    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.598809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.598722    5958 generic.go:356] "Generic (PLEG): container finished" podID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerID="ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511" exitCode=0
Oct 31 19:56:57.598809 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.598786    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"
Oct 31 19:56:57.599011 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.598802    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" event={"ID":"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4","Type":"ContainerDied","Data":"ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511"}
Oct 31 19:56:57.599011 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.598857    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6" event={"ID":"5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4","Type":"ContainerDied","Data":"b19d8e630b1953014aae9384ac25ea45ff3c4db41c912e9ca4c5a0d343b5e151"}
Oct 31 19:56:57.599011 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.598883    5958 scope.go:122] "RemoveContainer" containerID="ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511"
Oct 31 19:56:57.613293 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.613274    5958 scope.go:122] "RemoveContainer" containerID="ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511"
Oct 31 19:56:57.613701 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:57.613680    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511\": container with ID starting with ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511 not found: ID does not exist" containerID="ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511"
Oct 31 19:56:57.613770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.613710    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511"} err="failed to get container status \"ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511\": rpc error: code = NotFound desc = could not find container \"ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511\": container with ID starting with ff5096cf8b18bc8beb49d62b9c3d2985649926946534924ad644b381de737511 not found: ID does not exist"
Oct 31 19:56:57.634462 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.634413    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"]
Oct 31 19:56:57.640413 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:57.640381    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-xz8h6"]
Oct 31 19:56:58.159826 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.159799    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:56:58.235804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.235770    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.235845    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.235876    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.235920    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236029 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.235989    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236093    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236136    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236210    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236247    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:58.236514 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236314    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs\") pod \"97622e45-12bd-4731-882f-0de4f56a3aa7\" (UID: \"97622e45-12bd-4731-882f-0de4f56a3aa7\") "
Oct 31 19:56:58.236629 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236565    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.236778 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236720    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.236850 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236789    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.236981 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.236959    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.237276 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.237254    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.237380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.237283    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.237380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.237298    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97622e45-12bd-4731-882f-0de4f56a3aa7-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.237380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.237311    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.237380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.237324    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.239244 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.239212    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "kube-api-access-nwpf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:58.239533 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.239513    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:58.240890 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.240863    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:58.242472 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.242446    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy" pod "97622e45-12bd-4731-882f-0de4f56a3aa7" (UID: "97622e45-12bd-4731-882f-0de4f56a3aa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:58.350567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.345777    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.350567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.345820    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.350567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.345837    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.350567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.345854    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwpf4\" (UniqueName: \"kubernetes.io/projected/97622e45-12bd-4731-882f-0de4f56a3aa7-kube-api-access-nwpf4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:58.609015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.608927    5958 generic.go:356] "Generic (PLEG): container finished" podID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerID="9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0" exitCode=0
Oct 31 19:56:58.609015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.608976    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"
Oct 31 19:56:58.609015 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.609008    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" event={"ID":"97622e45-12bd-4731-882f-0de4f56a3aa7","Type":"ContainerDied","Data":"9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0"}
Oct 31 19:56:58.609277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.609064    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78" event={"ID":"97622e45-12bd-4731-882f-0de4f56a3aa7","Type":"ContainerDied","Data":"3116cb3e62e3e29dcd800aa43f057415d79a9551e40f395643dde75e05d2748f"}
Oct 31 19:56:58.609277 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.609091    5958 scope.go:122] "RemoveContainer" containerID="9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0"
Oct 31 19:56:58.621401 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.621379    5958 scope.go:122] "RemoveContainer" containerID="9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0"
Oct 31 19:56:58.621753 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:56:58.621731    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0\": container with ID starting with 9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0 not found: ID does not exist" containerID="9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0"
Oct 31 19:56:58.621818 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.621764    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0"} err="failed to get container status \"9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0\": rpc error: code = NotFound desc = could not find container \"9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0\": container with ID starting with 9a608787ecc2557d7ddfcd14cc0d7e630be2d8031c84f46af87ec2f9434402d0 not found: ID does not exist"
Oct 31 19:56:58.644636 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.644603    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"]
Oct 31 19:56:58.649935 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.649904    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-tbx78"]
Oct 31 19:56:58.667374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.667344    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" path="/var/lib/kubelet/pods/5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4/volumes"
Oct 31 19:56:58.667977 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:56:58.667961    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" path="/var/lib/kubelet/pods/97622e45-12bd-4731-882f-0de4f56a3aa7/volumes"
Oct 31 19:57:06.802203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:06.801759    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"]
Oct 31 19:57:06.802203 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:06.802061    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy" containerID="cri-o://0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1" gracePeriod=30
Oct 31 19:57:07.039463 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:07.039429    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"]
Oct 31 19:57:07.039723 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:07.039656    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" containerID="cri-o://c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af" gracePeriod=30
Oct 31 19:57:09.373361 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:09.373315    5958 scope.go:122] "RemoveContainer" containerID="fa51681f8ed25fdeac80e506de771f1bb705be0fede7ea0fe586eefcd0494548"
Oct 31 19:57:12.181699 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.181663    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:57:12.259190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259152    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259262    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259357 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259336    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259453 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259385    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259453 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259438    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259530 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259485    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259601 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259525    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259668 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259649    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259741    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo\") pod \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\" (UID: \"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466\") "
Oct 31 19:57:12.259804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.259785    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.260298 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.260272    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.260523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.260500    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.260523 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.260516    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.260746 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.260556    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.260813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.260772    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.260948 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.260922    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:12.263820 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.263791    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "kube-api-access-q9xsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:12.267721 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.267689    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:57:12.267856 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.267796    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:12.277718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.277685    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy" pod "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" (UID: "cbdec5e7-c13b-4bf1-9e24-b4d2585f2466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362221    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362268    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362283    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362297    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362313    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9xsb\" (UniqueName: \"kubernetes.io/projected/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-kube-api-access-q9xsb\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362328    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.363576 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.362341    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.407572 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.407521    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.463859    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.463938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.463980    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464032    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464121    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464175    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464382    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464446    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464493    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert\") pod \"2ff9953d-feea-480c-83d2-29225b3bf415\" (UID: \"2ff9953d-feea-480c-83d2-29225b3bf415\") "
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.464717    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.465596 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.465341    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.466632 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.466593    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:12.467131 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.467108    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.467295 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.467216    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.467504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.467481    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.467680 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.467570    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:12.469349 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.469322    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "kube-api-access-4ndvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:12.469650 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.469621    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:57:12.473755 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.473729    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token" pod "2ff9953d-feea-480c-83d2-29225b3bf415" (UID: "2ff9953d-feea-480c-83d2-29225b3bf415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:12.567307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567270    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567302    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9953d-feea-480c-83d2-29225b3bf415-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567307 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567315    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567324    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567333    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567342    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ndvr\" (UniqueName: \"kubernetes.io/projected/2ff9953d-feea-480c-83d2-29225b3bf415-kube-api-access-4ndvr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567351    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9953d-feea-480c-83d2-29225b3bf415-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.567630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.567361    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9953d-feea-480c-83d2-29225b3bf415-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:12.776379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.776335    5958 generic.go:356] "Generic (PLEG): container finished" podID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerID="0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1" exitCode=0
Oct 31 19:57:12.776574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.776400    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"
Oct 31 19:57:12.776574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.776420    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" event={"ID":"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466","Type":"ContainerDied","Data":"0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1"}
Oct 31 19:57:12.776574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.776520    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm" event={"ID":"cbdec5e7-c13b-4bf1-9e24-b4d2585f2466","Type":"ContainerDied","Data":"f0791e9cc87f6b7ea829d3b69db151ccd2bd5c34eed23a7b9310a6f02beeac82"}
Oct 31 19:57:12.776574 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.776564    5958 scope.go:122] "RemoveContainer" containerID="0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1"
Oct 31 19:57:12.778396 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.778368    5958 generic.go:356] "Generic (PLEG): container finished" podID="2ff9953d-feea-480c-83d2-29225b3bf415" containerID="c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af" exitCode=0
Oct 31 19:57:12.778511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.778429    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" event={"ID":"2ff9953d-feea-480c-83d2-29225b3bf415","Type":"ContainerDied","Data":"c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af"}
Oct 31 19:57:12.778511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.778435    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"
Oct 31 19:57:12.778511 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.778471    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" event={"ID":"2ff9953d-feea-480c-83d2-29225b3bf415","Type":"ContainerDied","Data":"39f4faec28131808450461a4e383678954ca9b753fea0532a390ecb2f6652f26"}
Oct 31 19:57:12.790327 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.790281    5958 scope.go:122] "RemoveContainer" containerID="0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1"
Oct 31 19:57:12.792690 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:57:12.792658    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1\": container with ID starting with 0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1 not found: ID does not exist" containerID="0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1"
Oct 31 19:57:12.792793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.792699    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1"} err="failed to get container status \"0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1\": rpc error: code = NotFound desc = could not find container \"0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1\": container with ID starting with 0e1226bacfd047e5e973e3e841f8fb456d661e9e17520158afdff293200c2af1 not found: ID does not exist"
Oct 31 19:57:12.792793 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.792723    5958 scope.go:122] "RemoveContainer" containerID="c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af"
Oct 31 19:57:12.803525 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.803491    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"]
Oct 31 19:57:12.804911 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.804884    5958 scope.go:122] "RemoveContainer" containerID="c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af"
Oct 31 19:57:12.805842 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:57:12.805807    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af\": container with ID starting with c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af not found: ID does not exist" containerID="c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af"
Oct 31 19:57:12.805971 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.805859    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af"} err="failed to get container status \"c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af\": rpc error: code = NotFound desc = could not find container \"c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af\": container with ID starting with c0ecc995b0393bdb08f56b97557245027cf6830dddea99636577fe30323711af not found: ID does not exist"
Oct 31 19:57:12.814900 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.814866    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-txzvm"]
Oct 31 19:57:12.834109 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.834077    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"]
Oct 31 19:57:12.840982 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:12.840952    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82"]
Oct 31 19:57:13.389391 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:13.389349    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:57:13.389941 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:13.389431    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bjc82" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:57:14.666399 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:14.666367    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" path="/var/lib/kubelet/pods/2ff9953d-feea-480c-83d2-29225b3bf415/volumes"
Oct 31 19:57:14.666971 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:14.666955    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" path="/var/lib/kubelet/pods/cbdec5e7-c13b-4bf1-9e24-b4d2585f2466/volumes"
Oct 31 19:57:17.047405 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:57:17.047356    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:57:44.749866 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.749826    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"]
Oct 31 19:57:44.751121 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751095    5958 cpu_manager.go:414] "containerMap: removing container" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy"
Oct 31 19:57:44.751215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751123    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy"
Oct 31 19:57:44.751215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751146    5958 cpu_manager.go:414] "containerMap: removing container" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerName="payload-pre-processing"
Oct 31 19:57:44.751215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751155    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerName="payload-pre-processing"
Oct 31 19:57:44.751215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751193    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerName="payload-processing"
Oct 31 19:57:44.751215 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751201    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerName="payload-processing"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751234    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751242    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751256    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751265    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751280    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751288    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751303    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751311    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751332    5958 cpu_manager.go:414] "containerMap: removing container" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy"
Oct 31 19:57:44.751405 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751340    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751600    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="97622e45-12bd-4731-882f-0de4f56a3aa7" containerName="istio-proxy"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751620    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="87973e9e-4ad8-48d1-8714-975d9bad5403" containerName="payload-pre-processing"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751636    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0381a0e-ac17-456e-81fc-d283cb5e62ae" containerName="payload-processing"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751650    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f3d6c3-bf3b-4570-a4ff-31c27131ade5" containerName="istio-proxy"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751660    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff9953d-feea-480c-83d2-29225b3bf415" containerName="istio-proxy"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751670    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f82a8eb-d0e5-47eb-9874-3d4ff936b5b4" containerName="istio-proxy"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751680    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f38528-4ce5-4c6f-8fb4-833bba442ff8" containerName="istio-proxy"
Oct 31 19:57:44.751823 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.751689    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdec5e7-c13b-4bf1-9e24-b4d2585f2466" containerName="istio-proxy"
Oct 31 19:57:44.756461 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.756439    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:44.768042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.768014    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"]
Oct 31 19:57:44.768042 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.768031    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1f2781-dockercfg-cx4jm\""
Oct 31 19:57:44.768208 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.768049    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1f2781\""
Oct 31 19:57:44.915571 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.913141    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"]
Oct 31 19:57:45.002750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.994799    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wn5l\" (UniqueName: \"kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.002750 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:44.994874    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.025830 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.025780    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"]
Oct 31 19:57:45.026267 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.026246    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.119459 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.119420    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.119717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.119539    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fwct\" (UniqueName: \"kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct\") pod \"payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.119717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.119596    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wn5l\" (UniqueName: \"kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.119717 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.119621    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.120619 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.120598    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.135010 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.134947    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wn5l\" (UniqueName: \"kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.221703 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.221664    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.221879 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.221774    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fwct\" (UniqueName: \"kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct\") pod \"payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.222415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.222396    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.235452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.235425    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fwct\" (UniqueName: \"kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct\") pod \"payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.357666 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.357581    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:45.393407 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.385264    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:45.730442 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:57:45.730403    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f8c36d_89fa_49ba_95c7_d8d9a429e76d.slice/crio-45d311185e0646daf59f55e535ff1dd54ee6dc765c42515e6f0157f554300a4f WatchSource:0}: Error finding container 45d311185e0646daf59f55e535ff1dd54ee6dc765c42515e6f0157f554300a4f: Status 404 returned error can't find the container with id 45d311185e0646daf59f55e535ff1dd54ee6dc765c42515e6f0157f554300a4f
Oct 31 19:57:45.730903 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.730875    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"]
Oct 31 19:57:45.732530 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:57:45.732416    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc49f2a6_8f3e_420b_b4d0_7fe5a6464253.slice/crio-6a817f234f36a69442d4058f091ef9f3a3cb5398efa08988defde1b2c4757c20 WatchSource:0}: Error finding container 6a817f234f36a69442d4058f091ef9f3a3cb5398efa08988defde1b2c4757c20: Status 404 returned error can't find the container with id 6a817f234f36a69442d4058f091ef9f3a3cb5398efa08988defde1b2c4757c20
Oct 31 19:57:45.732960 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.732943    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:57:45.735588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:45.735564    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"]
Oct 31 19:57:46.086610 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:46.086480    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj" event={"ID":"45f8c36d-89fa-49ba-95c7-d8d9a429e76d","Type":"ContainerStarted","Data":"45d311185e0646daf59f55e535ff1dd54ee6dc765c42515e6f0157f554300a4f"}
Oct 31 19:57:46.087980 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:46.087948    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq" event={"ID":"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253","Type":"ContainerStarted","Data":"6a817f234f36a69442d4058f091ef9f3a3cb5398efa08988defde1b2c4757c20"}
Oct 31 19:57:47.088018 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:57:47.087931    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:57:49.125133 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:49.125094    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj" event={"ID":"45f8c36d-89fa-49ba-95c7-d8d9a429e76d","Type":"ContainerStarted","Data":"b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf"}
Oct 31 19:57:49.125659 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:49.125173    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:57:49.127057 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:49.127024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq" event={"ID":"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253","Type":"ContainerStarted","Data":"df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba"}
Oct 31 19:57:49.127202 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:49.127105    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:57:49.149702 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:49.149649    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj" podStartSLOduration=2.862976432 podStartE2EDuration="5.149635872s" totalImagesPullingTime="2.28665944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:49.146406173 +0000 UTC m=+2262.607780789" watchObservedRunningTime="2026-10-31 19:57:49.149635872 +0000 UTC m=+2262.611010481"
Oct 31 19:57:49.170333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:57:49.170276    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq" podStartSLOduration=2.899057279 podStartE2EDuration="5.17026117s" totalImagesPullingTime="2.271203891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:49.166864359 +0000 UTC m=+2262.628238975" watchObservedRunningTime="2026-10-31 19:57:49.17026117 +0000 UTC m=+2262.631635785"
Oct 31 19:58:00.140122 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:00.140079    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 19:58:00.140858 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:00.140262    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 19:58:07.052505 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.052468    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"]
Oct 31 19:58:07.057390 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.057369    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.068834 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.068803    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f5c1ab-openshift-default-dockercfg-r9nbk\""
Oct 31 19:58:07.142594 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.126083    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"]
Oct 31 19:58:07.241933 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.241888    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.241979    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56pj\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242129 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242096    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242254 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242137    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242314 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242288    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242374 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242362    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242421 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242410    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242492 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242479    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.242599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.242526    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.343924    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344021 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.343993    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344042    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344090    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344185    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344251 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344228    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56pj\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344316    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344366    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344489 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344480    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344431    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344587    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.344748 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.344737    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.345046 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.345025    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.346763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.346739    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.346873 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.346791    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.357843 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.357815    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.357993 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.357813    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56pj\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.438718 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.438683    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:07.781145 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.781113    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"]
Oct 31 19:58:07.781625 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:58:07.781595    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5844ffbc_e285_4de7_9af7_b6a384bc398b.slice/crio-e05c5057cdb2b33cefb0493e2720207b06dc8785d07da8cba36fbf068b8f73af WatchSource:0}: Error finding container e05c5057cdb2b33cefb0493e2720207b06dc8785d07da8cba36fbf068b8f73af: Status 404 returned error can't find the container with id e05c5057cdb2b33cefb0493e2720207b06dc8785d07da8cba36fbf068b8f73af
Oct 31 19:58:07.784813 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.784776    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:58:07.784945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.784846    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:58:07.784945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:07.784875    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:58:08.333583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:08.333538    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" event={"ID":"5844ffbc-e285-4de7-9af7-b6a384bc398b","Type":"ContainerStarted","Data":"3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111"}
Oct 31 19:58:08.334007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:08.333604    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" event={"ID":"5844ffbc-e285-4de7-9af7-b6a384bc398b","Type":"ContainerStarted","Data":"e05c5057cdb2b33cefb0493e2720207b06dc8785d07da8cba36fbf068b8f73af"}
Oct 31 19:58:08.363988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:08.363928    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podStartSLOduration=1.3639113489999999 podStartE2EDuration="1.363911349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:08.362824526 +0000 UTC m=+2281.824199140" watchObservedRunningTime="2026-10-31 19:58:08.363911349 +0000 UTC m=+2281.825285964"
Oct 31 19:58:08.439340 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:08.439301    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:08.440559 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:08.440525    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:08.440683 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:08.440602    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:09.439097 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:09.439061    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:09.439597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:09.439134    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:10.439880 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:10.439847    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:10.440358 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:10.439905    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:11.439056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:11.439015    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:11.439333 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:11.439096    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:12.441854 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:12.441815    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:12.442524 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:12.442476    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:13.438956 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:13.438920    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:13.439151 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:13.438970    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:14.138847 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:14.138811    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 19:58:14.444207 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:14.444114    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Oct 31 19:58:14.444496 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:14.444467    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Oct 31 19:58:15.445181 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:15.445155    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:15.445684 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:15.445652    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:15.446916 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:15.446891    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 19:58:17.051129 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:58:17.051089    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:58:43.481180 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.481145    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"]
Oct 31 19:58:43.487148 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.487123    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.512025 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.511998    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-26cd59-serving-cert\""
Oct 31 19:58:43.521007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.520975    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"]
Oct 31 19:58:43.602560 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.602517    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.602832 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.602810    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.603007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.602987    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmt8f\" (UniqueName: \"kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.603056 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.603021    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.711065 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.711022    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.711677 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.711646    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.720654 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.712238    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmt8f\" (UniqueName: \"kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.720905 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.720884    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.721814 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.721742    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.727616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.727164    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.727616 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.727567    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.734617 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.734476    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmt8f\" (UniqueName: \"kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f\") pod \"maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:43.809328 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:43.809294    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:44.177387 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:44.177348    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"]
Oct 31 19:58:44.869368 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:44.869331    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk" event={"ID":"63ee7911-d643-47bd-bd64-031db7fdd5ae","Type":"ContainerStarted","Data":"7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5"}
Oct 31 19:58:44.869830 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:44.869396    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk" event={"ID":"63ee7911-d643-47bd-bd64-031db7fdd5ae","Type":"ContainerStarted","Data":"844dabb01e919c87dd8b86e02c8703c5edef45decec732fd5ca81ef3a0d2fabb"}
Oct 31 19:58:44.869830 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:44.869562    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:58:44.899125 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:44.899076    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk" podStartSLOduration=1.473325751 podStartE2EDuration="1.899060933s" totalImagesPullingTime="425.735182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:44.895084113 +0000 UTC m=+2318.356458730" watchObservedRunningTime="2026-10-31 19:58:44.899060933 +0000 UTC m=+2318.360435548"
Oct 31 19:58:47.064771 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:58:47.064728    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:58:50.883978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:58:50.883947    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 19:59:17.048869 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:59:17.048774    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:59:21.215670 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.215629    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"]
Oct 31 19:59:21.223959 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.223925    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.257945 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.257896    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"]
Oct 31 19:59:21.314696 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.314665    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-01afb5-openshift-default-dockercfg-czt6t\""
Oct 31 19:59:21.320379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320354    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2r5\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320515 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320491    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320568    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320689 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320677    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320729 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320703    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320800 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320785    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320860 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320825    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.320944 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320931    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.321002 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.320976    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423238 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423195    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423258    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423318    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423415 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423351    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423539 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423480    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423538    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423688 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423678    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2r5\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423802 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423778    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423806    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423849 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423793    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423872    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.423923 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.423890    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.424110 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.424089    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.424491 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.424473    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.426318 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.426296    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.426599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.426579    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.437528 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.437493    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.437973 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.437908    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2r5\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.565228 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.565153    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:21.859950 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:59:21.859918    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708a5ebd_a589_4796_84d3_26d27841278d.slice/crio-06cd2f8ca7bca5633742c32ac30d0f2e2d4324ad65cbadb1dfd8cfdf729c33d9 WatchSource:0}: Error finding container 06cd2f8ca7bca5633742c32ac30d0f2e2d4324ad65cbadb1dfd8cfdf729c33d9: Status 404 returned error can't find the container with id 06cd2f8ca7bca5633742c32ac30d0f2e2d4324ad65cbadb1dfd8cfdf729c33d9
Oct 31 19:59:21.860434 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.860409    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"]
Oct 31 19:59:21.863068 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.863034    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:21.863190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.863104    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:21.863190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:21.863134    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:22.501105 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:22.501063    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" event={"ID":"708a5ebd-a589-4796-84d3-26d27841278d","Type":"ContainerStarted","Data":"5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3"}
Oct 31 19:59:22.501625 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:22.501130    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" event={"ID":"708a5ebd-a589-4796-84d3-26d27841278d","Type":"ContainerStarted","Data":"06cd2f8ca7bca5633742c32ac30d0f2e2d4324ad65cbadb1dfd8cfdf729c33d9"}
Oct 31 19:59:22.536140 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:22.536083    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podStartSLOduration=1.53606856 podStartE2EDuration="1.53606856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:22.530943223 +0000 UTC m=+2355.992317850" watchObservedRunningTime="2026-10-31 19:59:22.53606856 +0000 UTC m=+2355.997443233"
Oct 31 19:59:22.565367 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:22.565332    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:22.566938 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:22.566906    5958 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:59:22.567100 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:22.566978    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:59:23.566081 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:23.566042    5958 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:59:23.566672 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:23.566101    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:59:24.566467 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:24.566425    5958 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:59:24.566955 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:24.566481    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:59:25.655570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:25.654174    5958 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:59:25.655570 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:25.654232    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:59:26.567588 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:26.567559    5958 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:59:26.567770 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:26.567611    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:59:27.566346 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:27.566306    5958 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:59:27.566801 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:27.566367    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 31 19:59:28.572383 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:28.572349    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:28.573742 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:28.573108    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:28.573887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:28.573855    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 19:59:37.373952 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.372634    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"]
Oct 31 19:59:37.380743 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.380681    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.390804 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.390773    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"]
Oct 31 19:59:37.467941 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.467906    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.467953    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468047    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468138 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468107    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468291 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468264    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468329 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468314    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468371 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468358    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946cm\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468472    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.468538 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.468527    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570272    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570322 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570325    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570386    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570446    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570539    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570597 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570591    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570782 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570623    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946cm\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.570894 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570867    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.571098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.570977    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.571098 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.571089    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.571280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.571147    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.571280 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.571199    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.571402 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.571352    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.571500 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.571482    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.573028 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.573005    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.573468 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.573449    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.584379 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.584354    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.584752 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.584729    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946cm\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:37.710263 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:37.710179    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:38.018887 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:59:38.017714    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5eec6e7_f7ed_4db8_ae0e_7a3e6958b1d3.slice/crio-54bc20850c7707232b799c37ddfa7020a7999b7c9833b0edd573e9ea3e1af57d WatchSource:0}: Error finding container 54bc20850c7707232b799c37ddfa7020a7999b7c9833b0edd573e9ea3e1af57d: Status 404 returned error can't find the container with id 54bc20850c7707232b799c37ddfa7020a7999b7c9833b0edd573e9ea3e1af57d
Oct 31 19:59:38.018887 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.017926    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"]
Oct 31 19:59:38.025119 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.025084    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:38.025246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.025155    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:38.025246 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.025186    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:38.906195 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.906144    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" event={"ID":"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3","Type":"ContainerStarted","Data":"9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1"}
Oct 31 19:59:38.906751 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.906218    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" event={"ID":"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3","Type":"ContainerStarted","Data":"54bc20850c7707232b799c37ddfa7020a7999b7c9833b0edd573e9ea3e1af57d"}
Oct 31 19:59:38.939053 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:38.938998    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podStartSLOduration=1.9389833319999998 podStartE2EDuration="1.938983332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:38.933863016 +0000 UTC m=+2372.395237635" watchObservedRunningTime="2026-10-31 19:59:38.938983332 +0000 UTC m=+2372.400357946"
Oct 31 19:59:39.715536 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:39.711063    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:39.759567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:39.752683    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 31 19:59:39.759567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:39.752748    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 31 19:59:40.722567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:40.715895    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 31 19:59:40.722567 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:40.715997    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 31 19:59:41.721965 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:41.721920    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 31 19:59:41.722190 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:41.721986    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 31 19:59:42.465484 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.465452    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"]
Oct 31 19:59:42.471452 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.471429    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.482598 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.481736    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-d9c048-openshift-default-dockercfg-fk2wl\""
Oct 31 19:59:42.487829 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.487798    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"]
Oct 31 19:59:42.665710 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.665658    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.665919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.665777    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.665919 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.665921    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.666051 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.665961    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.666132 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.666102    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.666222 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.666199    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.666269 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.666257    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.666320 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.666289    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.666459 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.666439    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck528\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.714007 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.713970    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 31 19:59:42.721630 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.714284    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 31 19:59:42.768032 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.767993    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768041    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768090    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768130    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768219 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768165    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768322    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768455 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768427    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck528\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768512    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768583 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768532    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768706 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768639    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768758 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768720    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768815 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768793    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.768978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.768956    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.769274 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.769251    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.770783 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.770758    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.771408 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.771386    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.781504 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.781478    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.782640 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.782619    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck528\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:42.791978 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:42.791951    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:43.137557 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.137508    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"]
Oct 31 19:59:43.138722 ip-10-0-138-35 kubenswrapper[5958]: W1031 19:59:43.138594    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9284c37_4d8c_4e02_985f_b63923457bc2.slice/crio-c9d6d1112c7146eaea8b01a6ea3a26b49423b2dd197e2701ca896829556bb6da WatchSource:0}: Error finding container c9d6d1112c7146eaea8b01a6ea3a26b49423b2dd197e2701ca896829556bb6da: Status 404 returned error can't find the container with id c9d6d1112c7146eaea8b01a6ea3a26b49423b2dd197e2701ca896829556bb6da
Oct 31 19:59:43.143255 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.143211    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:43.143393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.143314    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:43.143393 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.143359    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 19:59:43.712618 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.712519    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 31 19:59:43.716698 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.716656    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 31 19:59:43.993359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.993255    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" event={"ID":"c9284c37-4d8c-4e02-985f-b63923457bc2","Type":"ContainerStarted","Data":"df685ad9701762aea6e0ed9302038d29e11c0fe95b2d7823e89bf0be6e1f9d94"}
Oct 31 19:59:43.993359 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:43.993325    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" event={"ID":"c9284c37-4d8c-4e02-985f-b63923457bc2","Type":"ContainerStarted","Data":"c9d6d1112c7146eaea8b01a6ea3a26b49423b2dd197e2701ca896829556bb6da"}
Oct 31 19:59:44.045490 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:44.045413    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podStartSLOduration=2.045392326 podStartE2EDuration="2.045392326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:44.045018791 +0000 UTC m=+2377.506393407" watchObservedRunningTime="2026-10-31 19:59:44.045392326 +0000 UTC m=+2377.506766943"
Oct 31 19:59:44.734599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:44.732584    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 31 19:59:44.734599 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:44.732682    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 31 19:59:44.792998 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:44.792963    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:44.795763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:44.795037    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 19:59:44.795763 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:44.795097    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 19:59:45.793335 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:45.793292    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 19:59:45.793774 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:45.793368    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 19:59:45.947628 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:45.947591    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:45.948179 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:45.948158    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:45.949205 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:45.949187    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 19:59:46.792921 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:46.792882    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 19:59:46.793102 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:46.792947    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 19:59:47.061157 ip-10-0-138-35 kubenswrapper[5958]: E1031 19:59:47.061064    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 19:59:47.792766 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:47.792734    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 19:59:47.792988 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:47.792793    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 19:59:48.792501 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:48.792462    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 19:59:48.792913 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:48.792520    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 19:59:49.792641 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:49.792605    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Oct 31 19:59:49.793048 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:49.792661    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Oct 31 19:59:50.797976 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:50.797935    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:50.798480 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:50.798317    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:50.799060 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:50.799038    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 19:59:53.271380 ip-10-0-138-35 kubenswrapper[5958]: I1031 19:59:53.271341    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"]
Oct 31 20:00:00.341868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.341833    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"]
Oct 31 20:00:00.348881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.348852    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.373447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.371135    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"]
Oct 31 20:00:00.378503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.378427    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:00:00.381256 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.381227    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"]
Oct 31 20:00:00.387933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.387901    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.388110 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.388058    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4c5\" (UniqueName: \"kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.389436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.389419    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.404010 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.403848    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"]
Oct 31 20:00:00.489348 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.489308    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.489578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.489459    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tfp\" (UniqueName: \"kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.489748 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.489722    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.489842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.489826    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4c5\" (UniqueName: \"kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.490383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.490357    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.516103 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.516057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4c5\" (UniqueName: \"kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.522122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.522094    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"]
Oct 31 20:00:00.527511 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.527487    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.544130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.544083    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:00.548307 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.548280    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"]
Oct 31 20:00:00.591638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.591597    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.591849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.591706    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.591849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.591791    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tfp\" (UniqueName: \"kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.592286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.592199    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7g9q\" (UniqueName: \"kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.592463 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.592438    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.592578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.592511    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.607015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.606984    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tfp\" (UniqueName: \"kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.688743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.688710    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:00.693854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.693823    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7g9q\" (UniqueName: \"kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.694504 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.694478    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.694666 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.694555    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.695185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.695160    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.702558 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.702495    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.716415 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.716387    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7g9q\" (UniqueName: \"kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q\") pod \"maas-api-revoke-sub-9bbcbf96-5q6g7\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:00.719309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.719286    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:00.851897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:00.851814    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:01.000262 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:00:01.000219    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc683a98_4639_4d32_88de_9e7d32621b28.slice/crio-54b42453cebd79d553dbdd8aca024b94a0ac0037c297410a99b95d825e586124 WatchSource:0}: Error finding container 54b42453cebd79d553dbdd8aca024b94a0ac0037c297410a99b95d825e586124: Status 404 returned error can't find the container with id 54b42453cebd79d553dbdd8aca024b94a0ac0037c297410a99b95d825e586124
Oct 31 20:00:01.004748 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:01.003253    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"]
Oct 31 20:00:01.035069 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:00:01.035043    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0686fcd4_01d7_4e94_bbe2_ce36e5193e74.slice/crio-d79af5e6a757db95379386c53bee72eb10135a042be89e2f6f0d78f459926eb3 WatchSource:0}: Error finding container d79af5e6a757db95379386c53bee72eb10135a042be89e2f6f0d78f459926eb3: Status 404 returned error can't find the container with id d79af5e6a757db95379386c53bee72eb10135a042be89e2f6f0d78f459926eb3
Oct 31 20:00:01.042646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:01.039866    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"]
Oct 31 20:00:01.179365 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:01.179335    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"]
Oct 31 20:00:01.218591 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:00:01.210390    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fae7b9e_3622_4f2b_8126_c50b2b986187.slice/crio-52386a095ea601a593463c55cccb6cf245145ee31fc3d2e203420ccf6c4f27e2 WatchSource:0}: Error finding container 52386a095ea601a593463c55cccb6cf245145ee31fc3d2e203420ccf6c4f27e2: Status 404 returned error can't find the container with id 52386a095ea601a593463c55cccb6cf245145ee31fc3d2e203420ccf6c4f27e2
Oct 31 20:00:01.284999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:01.284959    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7" event={"ID":"5fae7b9e-3622-4f2b-8126-c50b2b986187","Type":"ContainerStarted","Data":"52386a095ea601a593463c55cccb6cf245145ee31fc3d2e203420ccf6c4f27e2"}
Oct 31 20:00:01.286889 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:01.286856    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerStarted","Data":"d79af5e6a757db95379386c53bee72eb10135a042be89e2f6f0d78f459926eb3"}
Oct 31 20:00:01.288581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:01.288556    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerStarted","Data":"54b42453cebd79d553dbdd8aca024b94a0ac0037c297410a99b95d825e586124"}
Oct 31 20:00:02.299975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.299885    5958 generic.go:356] "Generic (PLEG): container finished" podID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerID="aa2dc2b3fd63195d2d98fefdcd8084c935853ed9041f01565521731b182f7af1" exitCode=60
Oct 31 20:00:02.300408 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.299970    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerDied","Data":"aa2dc2b3fd63195d2d98fefdcd8084c935853ed9041f01565521731b182f7af1"}
Oct 31 20:00:02.300408 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.300360    5958 scope.go:122] "RemoveContainer" containerID="aa2dc2b3fd63195d2d98fefdcd8084c935853ed9041f01565521731b182f7af1"
Oct 31 20:00:02.301862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.301836    5958 generic.go:356] "Generic (PLEG): container finished" podID="dc683a98-4639-4d32-88de-9e7d32621b28" containerID="beded2f7ca6aa5c112924460ef753e466bc6a7ec31d24eff3e939038d6999e7a" exitCode=60
Oct 31 20:00:02.301968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.301909    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerDied","Data":"beded2f7ca6aa5c112924460ef753e466bc6a7ec31d24eff3e939038d6999e7a"}
Oct 31 20:00:02.302203 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.302185    5958 scope.go:122] "RemoveContainer" containerID="beded2f7ca6aa5c112924460ef753e466bc6a7ec31d24eff3e939038d6999e7a"
Oct 31 20:00:02.304129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.303958    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7" event={"ID":"5fae7b9e-3622-4f2b-8126-c50b2b986187","Type":"ContainerStarted","Data":"8cb6af90701cc60deab8bed7aa79e8501ed635251963b7b8fc2187e14a7c147f"}
Oct 31 20:00:02.409089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:02.409018    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7" podStartSLOduration=2.40899617 podStartE2EDuration="2.40899617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.405014606 +0000 UTC m=+2395.866389221" watchObservedRunningTime="2026-10-31 20:00:02.40899617 +0000 UTC m=+2395.870370790"
Oct 31 20:00:03.314091 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:03.313971    5958 generic.go:356] "Generic (PLEG): container finished" podID="5fae7b9e-3622-4f2b-8126-c50b2b986187" containerID="8cb6af90701cc60deab8bed7aa79e8501ed635251963b7b8fc2187e14a7c147f" exitCode=0
Oct 31 20:00:03.314091 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:03.314047    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7" event={"ID":"5fae7b9e-3622-4f2b-8126-c50b2b986187","Type":"ContainerDied","Data":"8cb6af90701cc60deab8bed7aa79e8501ed635251963b7b8fc2187e14a7c147f"}
Oct 31 20:00:03.316317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:03.316285    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerStarted","Data":"af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17"}
Oct 31 20:00:03.318278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:03.318255    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerStarted","Data":"fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68"}
Oct 31 20:00:03.390146 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:03.390087    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" podStartSLOduration=3.390072339 podStartE2EDuration="3.390072339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.386999914 +0000 UTC m=+2396.848374551" watchObservedRunningTime="2026-10-31 20:00:03.390072339 +0000 UTC m=+2396.851447044"
Oct 31 20:00:03.430649 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:03.430587    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" podStartSLOduration=3.430566934 podStartE2EDuration="3.430566934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.425177784 +0000 UTC m=+2396.886552400" watchObservedRunningTime="2026-10-31 20:00:03.430566934 +0000 UTC m=+2396.891941557"
Oct 31 20:00:04.330226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.330190    5958 generic.go:356] "Generic (PLEG): container finished" podID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerID="af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17" exitCode=60
Oct 31 20:00:04.330784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.330265    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerDied","Data":"af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17"}
Oct 31 20:00:04.330784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.330328    5958 scope.go:122] "RemoveContainer" containerID="aa2dc2b3fd63195d2d98fefdcd8084c935853ed9041f01565521731b182f7af1"
Oct 31 20:00:04.330784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.330728    5958 scope.go:122] "RemoveContainer" containerID="af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17"
Oct 31 20:00:04.331159 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:04.331137    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz_odh-ai-gateway-infra(0686fcd4-01d7-4e94-bbe2-ce36e5193e74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74"
Oct 31 20:00:04.332591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.332567    5958 generic.go:356] "Generic (PLEG): container finished" podID="dc683a98-4639-4d32-88de-9e7d32621b28" containerID="fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68" exitCode=60
Oct 31 20:00:04.332695 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.332627    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerDied","Data":"fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68"}
Oct 31 20:00:04.333010 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.332991    5958 scope.go:122] "RemoveContainer" containerID="fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68"
Oct 31 20:00:04.333595 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:04.333481    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w_odh-ai-gateway-infra(dc683a98-4639-4d32-88de-9e7d32621b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" podUID="dc683a98-4639-4d32-88de-9e7d32621b28"
Oct 31 20:00:04.349068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.349027    5958 scope.go:122] "RemoveContainer" containerID="beded2f7ca6aa5c112924460ef753e466bc6a7ec31d24eff3e939038d6999e7a"
Oct 31 20:00:04.595199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.593640    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:04.660722 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.660688    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca\") pod \"5fae7b9e-3622-4f2b-8126-c50b2b986187\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") "
Oct 31 20:00:04.660921 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.660804    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca\") pod \"5fae7b9e-3622-4f2b-8126-c50b2b986187\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") "
Oct 31 20:00:04.660989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.660968    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q\" (UniqueName: \"kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q\") pod \"5fae7b9e-3622-4f2b-8126-c50b2b986187\" (UID: \"5fae7b9e-3622-4f2b-8126-c50b2b986187\") "
Oct 31 20:00:04.661485 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.661364    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca" pod "5fae7b9e-3622-4f2b-8126-c50b2b986187" (UID: "5fae7b9e-3622-4f2b-8126-c50b2b986187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:04.661837 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.661811    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:04.663882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.663852    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca" pod "5fae7b9e-3622-4f2b-8126-c50b2b986187" (UID: "5fae7b9e-3622-4f2b-8126-c50b2b986187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:04.665646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.665615    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q" pod "5fae7b9e-3622-4f2b-8126-c50b2b986187" (UID: "5fae7b9e-3622-4f2b-8126-c50b2b986187"). InnerVolumeSpecName "kube-api-access-f7g9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:04.764236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.764194    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7g9q\" (UniqueName: \"kubernetes.io/projected/5fae7b9e-3622-4f2b-8126-c50b2b986187-kube-api-access-f7g9q\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:04.764236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:04.764233    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fae7b9e-3622-4f2b-8126-c50b2b986187-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:05.343038 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:05.343008    5958 scope.go:122] "RemoveContainer" containerID="af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17"
Oct 31 20:00:05.343594 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:05.343487    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz_odh-ai-gateway-infra(0686fcd4-01d7-4e94-bbe2-ce36e5193e74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74"
Oct 31 20:00:05.344826 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:05.344802    5958 scope.go:122] "RemoveContainer" containerID="fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68"
Oct 31 20:00:05.345231 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:05.345205    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w_odh-ai-gateway-infra(dc683a98-4639-4d32-88de-9e7d32621b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" podUID="dc683a98-4639-4d32-88de-9e7d32621b28"
Oct 31 20:00:05.346089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:05.346063    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"
Oct 31 20:00:05.346183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:05.346108    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7" event={"ID":"5fae7b9e-3622-4f2b-8126-c50b2b986187","Type":"ContainerDied","Data":"52386a095ea601a593463c55cccb6cf245145ee31fc3d2e203420ccf6c4f27e2"}
Oct 31 20:00:05.346183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:05.346156    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52386a095ea601a593463c55cccb6cf245145ee31fc3d2e203420ccf6c4f27e2"
Oct 31 20:00:06.282333 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.282294    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"]
Oct 31 20:00:06.283905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.283847    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5fae7b9e-3622-4f2b-8126-c50b2b986187" containerName="revoke-keys"
Oct 31 20:00:06.283905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.283873    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fae7b9e-3622-4f2b-8126-c50b2b986187" containerName="revoke-keys"
Oct 31 20:00:06.284267 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.284194    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fae7b9e-3622-4f2b-8126-c50b2b986187" containerName="revoke-keys"
Oct 31 20:00:06.293078 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.290671    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.304865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.304832    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"]
Oct 31 20:00:06.308801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.308777    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5f3eff-serving-cert\""
Oct 31 20:00:06.395111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.389526    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.395111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.389723    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.395111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.389984    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.395111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.390030    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j82f\" (UniqueName: \"kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.492089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.492054    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.492254 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.492107    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j82f\" (UniqueName: \"kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.492254 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.492216    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.492356 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.492267    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.493044 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.493016    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.494725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.494700    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.494862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.494838    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.514012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.513983    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j82f\" (UniqueName: \"kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f\") pod \"maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.612925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.612848    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:06.913974 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:00:06.913941    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb43a3d_0cd6_4ed5_9227_dfbc759be3d6.slice/crio-98737962302f2352bf099da149bd3ae26bf1ff17d886f7f564e24f99de3ee331 WatchSource:0}: Error finding container 98737962302f2352bf099da149bd3ae26bf1ff17d886f7f564e24f99de3ee331: Status 404 returned error can't find the container with id 98737962302f2352bf099da149bd3ae26bf1ff17d886f7f564e24f99de3ee331
Oct 31 20:00:06.914814 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:06.914784    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"]
Oct 31 20:00:07.333303 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:07.333225    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"]
Oct 31 20:00:07.333481 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:07.333448    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj" podUID="45f8c36d-89fa-49ba-95c7-d8d9a429e76d" containerName="payload-processing" containerID="cri-o://b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf" gracePeriod=30
Oct 31 20:00:07.382302 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:07.379581    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"]
Oct 31 20:00:07.387479 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:07.385825    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq" podUID="fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" containerName="payload-pre-processing" containerID="cri-o://df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba" gracePeriod=30
Oct 31 20:00:07.401227 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:07.400866    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll" event={"ID":"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6","Type":"ContainerStarted","Data":"98737962302f2352bf099da149bd3ae26bf1ff17d886f7f564e24f99de3ee331"}
Oct 31 20:00:07.980176 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:07.980150    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 20:00:08.063219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.063166    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct\" (UniqueName: \"kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct\") pod \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") "
Oct 31 20:00:08.063581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.063536    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume\") pod \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\" (UID: \"45f8c36d-89fa-49ba-95c7-d8d9a429e76d\") "
Oct 31 20:00:08.065534 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.065500    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume" pod "45f8c36d-89fa-49ba-95c7-d8d9a429e76d" (UID: "45f8c36d-89fa-49ba-95c7-d8d9a429e76d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:08.070938 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.070895    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct" pod "45f8c36d-89fa-49ba-95c7-d8d9a429e76d" (UID: "45f8c36d-89fa-49ba-95c7-d8d9a429e76d"). InnerVolumeSpecName "kube-api-access-7fwct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:08.137079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.137009    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 20:00:08.166114 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.166072    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l\" (UniqueName: \"kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l\") pod \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") "
Oct 31 20:00:08.166303 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.166175    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume\") pod \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\" (UID: \"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253\") "
Oct 31 20:00:08.167230 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.166891    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume" pod "fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" (UID: "fc49f2a6-8f3e-420b-b4d0-7fe5a6464253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:08.167466 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.167341    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fwct\" (UniqueName: \"kubernetes.io/projected/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-kube-api-access-7fwct\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.167466 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.167368    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f8c36d-89fa-49ba-95c7-d8d9a429e76d-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.167466 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.167379    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.168936 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.168909    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l" pod "fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" (UID: "fc49f2a6-8f3e-420b-b4d0-7fe5a6464253"). InnerVolumeSpecName "kube-api-access-4wn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:08.269703 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.269665    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wn5l\" (UniqueName: \"kubernetes.io/projected/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253-kube-api-access-4wn5l\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.323608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.323534    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" podUID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" containerName="main" containerID="cri-o://d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0" gracePeriod=45
Oct 31 20:00:08.411753 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.411721    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" containerID="df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba" exitCode=0
Oct 31 20:00:08.412242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.411773    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"
Oct 31 20:00:08.412242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.411805    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq" event={"ID":"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253","Type":"ContainerDied","Data":"df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba"}
Oct 31 20:00:08.412242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.411899    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq" event={"ID":"fc49f2a6-8f3e-420b-b4d0-7fe5a6464253","Type":"ContainerDied","Data":"6a817f234f36a69442d4058f091ef9f3a3cb5398efa08988defde1b2c4757c20"}
Oct 31 20:00:08.412242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.411930    5958 scope.go:122] "RemoveContainer" containerID="df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba"
Oct 31 20:00:08.413954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.413928    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll" event={"ID":"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6","Type":"ContainerStarted","Data":"ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd"}
Oct 31 20:00:08.414079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.414045    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:08.415696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.415673    5958 generic.go:356] "Generic (PLEG): container finished" podID="45f8c36d-89fa-49ba-95c7-d8d9a429e76d" containerID="b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf" exitCode=0
Oct 31 20:00:08.415791 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.415740    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"
Oct 31 20:00:08.415791 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.415764    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj" event={"ID":"45f8c36d-89fa-49ba-95c7-d8d9a429e76d","Type":"ContainerDied","Data":"b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf"}
Oct 31 20:00:08.415901 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.415797    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj" event={"ID":"45f8c36d-89fa-49ba-95c7-d8d9a429e76d","Type":"ContainerDied","Data":"45d311185e0646daf59f55e535ff1dd54ee6dc765c42515e6f0157f554300a4f"}
Oct 31 20:00:08.438927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.438901    5958 scope.go:122] "RemoveContainer" containerID="df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba"
Oct 31 20:00:08.439288 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:08.439267    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba\": container with ID starting with df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba not found: ID does not exist" containerID="df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba"
Oct 31 20:00:08.439346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.439299    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba"} err="failed to get container status \"df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba\": rpc error: code = NotFound desc = could not find container \"df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba\": container with ID starting with df256c9cd438e22c6b2eb2243ead5254e1cd33b5fd3cacf81fd1c9b86e7010ba not found: ID does not exist"
Oct 31 20:00:08.439346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.439314    5958 scope.go:122] "RemoveContainer" containerID="b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf"
Oct 31 20:00:08.446348 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.446309    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" podUID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.204:8000/ready\": dial tcp 10.134.0.204:8000: connect: connection refused"
Oct 31 20:00:08.448632 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.448591    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll" podStartSLOduration=1.996477327 podStartE2EDuration="2.448576695s" totalImagesPullingTime="452.099368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:08.443418488 +0000 UTC m=+2401.904793096" watchObservedRunningTime="2026-10-31 20:00:08.448576695 +0000 UTC m=+2401.909951310"
Oct 31 20:00:08.456491 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.456469    5958 scope.go:122] "RemoveContainer" containerID="b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf"
Oct 31 20:00:08.456941 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:08.456922    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf\": container with ID starting with b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf not found: ID does not exist" containerID="b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf"
Oct 31 20:00:08.457013 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.456949    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf"} err="failed to get container status \"b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf\": rpc error: code = NotFound desc = could not find container \"b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf\": container with ID starting with b75d478bbe1b2bb10fcb89f5d677306cd1528a6f5e284e595bbe3044f47048cf not found: ID does not exist"
Oct 31 20:00:08.471483 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.471069    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"]
Oct 31 20:00:08.482262 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.482231    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-5d9d88cccf-2pbxq"]
Oct 31 20:00:08.496616 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.496579    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"]
Oct 31 20:00:08.501084 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.501052    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-857bf89d44-gqflj"]
Oct 31 20:00:08.683815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.683746    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f8c36d-89fa-49ba-95c7-d8d9a429e76d" path="/var/lib/kubelet/pods/45f8c36d-89fa-49ba-95c7-d8d9a429e76d/volumes"
Oct 31 20:00:08.684283 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.684268    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" path="/var/lib/kubelet/pods/fc49f2a6-8f3e-420b-b4d0-7fe5a6464253/volumes"
Oct 31 20:00:08.722076 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.722048    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 20:00:08.807286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.807244    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir\") pod \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") "
Oct 31 20:00:08.807286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.807297    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq\" (UniqueName: \"kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq\") pod \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") "
Oct 31 20:00:08.807777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.807346    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache\") pod \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") "
Oct 31 20:00:08.807777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.807426    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs\") pod \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") "
Oct 31 20:00:08.807777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.807509    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm\") pod \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") "
Oct 31 20:00:08.807777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.807729    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home\") pod \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\" (UID: \"b5a7c51b-ab5b-4557-baa7-29a70955e3cb\") "
Oct 31 20:00:08.808815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.808788    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir" pod "b5a7c51b-ab5b-4557-baa7-29a70955e3cb" (UID: "b5a7c51b-ab5b-4557-baa7-29a70955e3cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:08.808994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.808905    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home" pod "b5a7c51b-ab5b-4557-baa7-29a70955e3cb" (UID: "b5a7c51b-ab5b-4557-baa7-29a70955e3cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:08.809146 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.809128    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache" pod "b5a7c51b-ab5b-4557-baa7-29a70955e3cb" (UID: "b5a7c51b-ab5b-4557-baa7-29a70955e3cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:08.812131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.812046    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm" pod "b5a7c51b-ab5b-4557-baa7-29a70955e3cb" (UID: "b5a7c51b-ab5b-4557-baa7-29a70955e3cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:08.812131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.812093    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq" pod "b5a7c51b-ab5b-4557-baa7-29a70955e3cb" (UID: "b5a7c51b-ab5b-4557-baa7-29a70955e3cb"). InnerVolumeSpecName "kube-api-access-576mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:08.813035 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.813000    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs" pod "b5a7c51b-ab5b-4557-baa7-29a70955e3cb" (UID: "b5a7c51b-ab5b-4557-baa7-29a70955e3cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:08.910659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.910619    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.910659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.910653    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.910659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.910667    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576mq\" (UniqueName: \"kubernetes.io/projected/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-kube-api-access-576mq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.910931 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.910682    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.910931 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.910696    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.910931 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:08.910709    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a7c51b-ab5b-4557-baa7-29a70955e3cb-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:09.428060 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.428023    5958 generic.go:356] "Generic (PLEG): container finished" podID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" containerID="d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0" exitCode=0
Oct 31 20:00:09.428610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.428080    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"
Oct 31 20:00:09.428610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.428105    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" event={"ID":"b5a7c51b-ab5b-4557-baa7-29a70955e3cb","Type":"ContainerDied","Data":"d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0"}
Oct 31 20:00:09.428610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.428148    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs" event={"ID":"b5a7c51b-ab5b-4557-baa7-29a70955e3cb","Type":"ContainerDied","Data":"bc50d0c000eddde4129afebe3e6dfdafc24d16b71477610576faf1654c5b46c9"}
Oct 31 20:00:09.428610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.428165    5958 scope.go:122] "RemoveContainer" containerID="d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0"
Oct 31 20:00:09.441456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.441438    5958 scope.go:122] "RemoveContainer" containerID="d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0"
Oct 31 20:00:09.441830 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:09.441801    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0\": container with ID starting with d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0 not found: ID does not exist" containerID="d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0"
Oct 31 20:00:09.441933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.441843    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0"} err="failed to get container status \"d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0\": rpc error: code = NotFound desc = could not find container \"d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0\": container with ID starting with d783fa8a10f2466d48b48deac1ac09154641dc98b21999a04fe89a10d68770b0 not found: ID does not exist"
Oct 31 20:00:09.456217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.456183    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"]
Oct 31 20:00:09.460977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:09.460944    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae4ce2/auth-test-model-ae4ce2-kserve-85fc5bb789-gqzjs"]
Oct 31 20:00:10.670189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:10.670158    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" path="/var/lib/kubelet/pods/b5a7c51b-ab5b-4557-baa7-29a70955e3cb/volumes"
Oct 31 20:00:14.436852 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:14.436821    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:00:14.656513 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:14.656476    5958 scope.go:122] "RemoveContainer" containerID="af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17"
Oct 31 20:00:15.486786 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:15.486683    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerStarted","Data":"0325adf02c9c2b447ad8315eb582e2bf4542d56960ca02e9aca5043128c2d368"}
Oct 31 20:00:16.497391 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:16.497355    5958 generic.go:356] "Generic (PLEG): container finished" podID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerID="0325adf02c9c2b447ad8315eb582e2bf4542d56960ca02e9aca5043128c2d368" exitCode=60
Oct 31 20:00:16.497976 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:16.497423    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerDied","Data":"0325adf02c9c2b447ad8315eb582e2bf4542d56960ca02e9aca5043128c2d368"}
Oct 31 20:00:16.497976 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:16.497489    5958 scope.go:122] "RemoveContainer" containerID="af80b0f200c9ba675cf0ef8561a942c0dcc8817edb26da23055cdfe573abdc17"
Oct 31 20:00:16.498101 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:16.497985    5958 scope.go:122] "RemoveContainer" containerID="0325adf02c9c2b447ad8315eb582e2bf4542d56960ca02e9aca5043128c2d368"
Oct 31 20:00:16.498423 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:16.498393    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz_odh-ai-gateway-infra(0686fcd4-01d7-4e94-bbe2-ce36e5193e74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74"
Oct 31 20:00:16.533175 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:16.533141    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"]
Oct 31 20:00:17.047969 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:17.047927    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:00:17.672568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:17.670119    5958 scope.go:122] "RemoveContainer" containerID="fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68"
Oct 31 20:00:18.050386 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.050359    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:18.190731 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.190700    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca\") pod \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") "
Oct 31 20:00:18.190916 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.190876    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp\" (UniqueName: \"kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp\") pod \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\" (UID: \"0686fcd4-01d7-4e94-bbe2-ce36e5193e74\") "
Oct 31 20:00:18.191167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.191139    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca" pod "0686fcd4-01d7-4e94-bbe2-ce36e5193e74" (UID: "0686fcd4-01d7-4e94-bbe2-ce36e5193e74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:18.191715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.191691    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.193139 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.193115    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp" pod "0686fcd4-01d7-4e94-bbe2-ce36e5193e74" (UID: "0686fcd4-01d7-4e94-bbe2-ce36e5193e74"). InnerVolumeSpecName "kube-api-access-96tfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:18.293714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.293630    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96tfp\" (UniqueName: \"kubernetes.io/projected/0686fcd4-01d7-4e94-bbe2-ce36e5193e74-kube-api-access-96tfp\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.307413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.307373    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:00:18.519304 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.519262    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz" event={"ID":"0686fcd4-01d7-4e94-bbe2-ce36e5193e74","Type":"ContainerDied","Data":"d79af5e6a757db95379386c53bee72eb10135a042be89e2f6f0d78f459926eb3"}
Oct 31 20:00:18.519513 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.519346    5958 scope.go:122] "RemoveContainer" containerID="0325adf02c9c2b447ad8315eb582e2bf4542d56960ca02e9aca5043128c2d368"
Oct 31 20:00:18.519513 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.519270    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"
Oct 31 20:00:18.521896 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.521868    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerStarted","Data":"0b509b740af938d826765826a9b23a2692472d785a97be50b8b01d72d9afdcf5"}
Oct 31 20:00:18.600503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.600420    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"]
Oct 31 20:00:18.622813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.622777    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891280-ld8dz"]
Oct 31 20:00:18.668367 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:18.668332    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" path="/var/lib/kubelet/pods/0686fcd4-01d7-4e94-bbe2-ce36e5193e74/volumes"
Oct 31 20:00:19.541358 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:19.541328    5958 generic.go:356] "Generic (PLEG): container finished" podID="dc683a98-4639-4d32-88de-9e7d32621b28" containerID="0b509b740af938d826765826a9b23a2692472d785a97be50b8b01d72d9afdcf5" exitCode=60
Oct 31 20:00:19.541897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:19.541395    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerDied","Data":"0b509b740af938d826765826a9b23a2692472d785a97be50b8b01d72d9afdcf5"}
Oct 31 20:00:19.541897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:19.541455    5958 scope.go:122] "RemoveContainer" containerID="fa761e8e1916abc59315dfb7c2356c459dcc260a71bf657256052414681bfc68"
Oct 31 20:00:19.542041 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:19.542022    5958 scope.go:122] "RemoveContainer" containerID="0b509b740af938d826765826a9b23a2692472d785a97be50b8b01d72d9afdcf5"
Oct 31 20:00:19.542786 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:19.542464    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w_odh-ai-gateway-infra(dc683a98-4639-4d32-88de-9e7d32621b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" podUID="dc683a98-4639-4d32-88de-9e7d32621b28"
Oct 31 20:00:19.586493 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:19.586183    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"]
Oct 31 20:00:20.821900 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.821816    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:20.885527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.885487    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"]
Oct 31 20:00:20.886857 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886830    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:20.886857 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886853    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886869    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886877    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886894    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886901    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886935    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886942    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886956    5958 cpu_manager.go:414] "containerMap: removing container" podUID="45f8c36d-89fa-49ba-95c7-d8d9a429e76d" containerName="payload-processing"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886963    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f8c36d-89fa-49ba-95c7-d8d9a429e76d" containerName="payload-processing"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886992    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" containerName="main"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.886999    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" containerName="main"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887020    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" containerName="payload-pre-processing"
Oct 31 20:00:20.887129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887027    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" containerName="payload-pre-processing"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887220    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887232    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887245    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887254    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887266    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f8c36d-89fa-49ba-95c7-d8d9a429e76d" containerName="payload-processing"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887279    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a7c51b-ab5b-4557-baa7-29a70955e3cb" containerName="main"
Oct 31 20:00:20.887881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.887289    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc49f2a6-8f3e-420b-b4d0-7fe5a6464253" containerName="payload-pre-processing"
Oct 31 20:00:20.919661 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.916226    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:20.927734 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.927685    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c734f5-dockercfg-kwbvg\""
Oct 31 20:00:20.943149 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.943110    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"]
Oct 31 20:00:20.944316 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.943535    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c734f5\""
Oct 31 20:00:20.993141 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.993110    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5\" (UniqueName: \"kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5\") pod \"dc683a98-4639-4d32-88de-9e7d32621b28\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") "
Oct 31 20:00:20.993337 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.993261    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca\") pod \"dc683a98-4639-4d32-88de-9e7d32621b28\" (UID: \"dc683a98-4639-4d32-88de-9e7d32621b28\") "
Oct 31 20:00:20.993664 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.993639    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca" pod "dc683a98-4639-4d32-88de-9e7d32621b28" (UID: "dc683a98-4639-4d32-88de-9e7d32621b28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:20.993818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.993803    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc683a98-4639-4d32-88de-9e7d32621b28-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.995501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:20.995470    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5" pod "dc683a98-4639-4d32-88de-9e7d32621b28" (UID: "dc683a98-4639-4d32-88de-9e7d32621b28"). InnerVolumeSpecName "kube-api-access-xk4c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:21.109691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.104894    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.109691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.105071    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5b4m\" (UniqueName: \"kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.109691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.105180    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4c5\" (UniqueName: \"kubernetes.io/projected/dc683a98-4639-4d32-88de-9e7d32621b28-kube-api-access-xk4c5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.209555 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.206796    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.209555 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.206961    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5b4m\" (UniqueName: \"kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.209824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.209614    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.234273 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.234179    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5b4m\" (UniqueName: \"kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.253820 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.253782    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:21.572568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.572104    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w" event={"ID":"dc683a98-4639-4d32-88de-9e7d32621b28","Type":"ContainerDied","Data":"54b42453cebd79d553dbdd8aca024b94a0ac0037c297410a99b95d825e586124"}
Oct 31 20:00:21.572568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.572191    5958 scope.go:122] "RemoveContainer" containerID="0b509b740af938d826765826a9b23a2692472d785a97be50b8b01d72d9afdcf5"
Oct 31 20:00:21.572568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.572335    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"
Oct 31 20:00:21.597741 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.597707    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"]
Oct 31 20:00:21.673331 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.673293    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"]
Oct 31 20:00:21.686506 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:21.686460    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891280-kg28w"]
Oct 31 20:00:22.584328 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:22.584292    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx" event={"ID":"28dc036c-2b1c-4f95-8929-b3e0f58c2300","Type":"ContainerStarted","Data":"7a873794beec0bc0f8586b4cc36d6f8f2b0d936c5a4245f198094122d70245c4"}
Oct 31 20:00:22.584850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:22.584355    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:22.584850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:22.584379    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx" event={"ID":"28dc036c-2b1c-4f95-8929-b3e0f58c2300","Type":"ContainerStarted","Data":"2ae2212d677efc5833bdacd417041a17a20e14d66c82b03204cbba802e212b02"}
Oct 31 20:00:22.670923 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:22.670859    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx" podStartSLOduration=2.670842613 podStartE2EDuration="2.670842613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:22.67045437 +0000 UTC m=+2416.131828986" watchObservedRunningTime="2026-10-31 20:00:22.670842613 +0000 UTC m=+2416.132217238"
Oct 31 20:00:22.678785 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:22.678751    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" path="/var/lib/kubelet/pods/dc683a98-4639-4d32-88de-9e7d32621b28/volumes"
Oct 31 20:00:23.226591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.226517    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"]
Oct 31 20:00:23.228075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.228048    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:23.228192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.228077    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:23.228192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.228116    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:23.228192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.228124    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:23.228429 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.228415    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0686fcd4-01d7-4e94-bbe2-ce36e5193e74" containerName="cleanup"
Oct 31 20:00:23.228462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.228437    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc683a98-4639-4d32-88de-9e7d32621b28" containerName="cleanup"
Oct 31 20:00:23.234393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.234370    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.252333 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.252298    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c173d8-serving-cert\""
Oct 31 20:00:23.258148 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.258114    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"]
Oct 31 20:00:23.310555 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.310495    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.310754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.310577    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.310899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.310879    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5r9w\" (UniqueName: \"kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.311030 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.311008    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.412606 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.412573    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.412834 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.412625    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.412918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.412893    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5r9w\" (UniqueName: \"kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.413375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.413348    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.413966 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.413741    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.415882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.415856    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.416932 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.416906    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.427377 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.427340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5r9w\" (UniqueName: \"kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w\") pod \"maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.569087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.568588    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:23.920008 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:00:23.919963    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d69ddb_4f71_45e8_8baf_b676887a2386.slice/crio-87fe4c259afd82874fbc185cdd44021fa9bbdd7e6e1b26d73ea57dd9570c04b8 WatchSource:0}: Error finding container 87fe4c259afd82874fbc185cdd44021fa9bbdd7e6e1b26d73ea57dd9570c04b8: Status 404 returned error can't find the container with id 87fe4c259afd82874fbc185cdd44021fa9bbdd7e6e1b26d73ea57dd9570c04b8
Oct 31 20:00:23.921437 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:23.921409    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"]
Oct 31 20:00:24.622424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:24.622366    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj" event={"ID":"a5d69ddb-4f71-45e8-8baf-b676887a2386","Type":"ContainerStarted","Data":"e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef"}
Oct 31 20:00:24.622702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:24.622429    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj" event={"ID":"a5d69ddb-4f71-45e8-8baf-b676887a2386","Type":"ContainerStarted","Data":"87fe4c259afd82874fbc185cdd44021fa9bbdd7e6e1b26d73ea57dd9570c04b8"}
Oct 31 20:00:24.622702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:24.622472    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:24.651517 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:24.651465    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj" podStartSLOduration=1.114305874 podStartE2EDuration="1.651449403s" totalImagesPullingTime="537.143529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:24.645975652 +0000 UTC m=+2418.107350268" watchObservedRunningTime="2026-10-31 20:00:24.651449403 +0000 UTC m=+2418.112824017"
Oct 31 20:00:30.639004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:30.638969    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:00:33.614975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:33.614940    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:00:43.707211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.707164    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"]
Oct 31 20:00:43.735069 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.734776    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"]
Oct 31 20:00:43.735069 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.734907    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.741918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.741493    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c734f5-serving-cert\""
Oct 31 20:00:43.824790 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.824749    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.825021 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.824994    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mws\" (UniqueName: \"kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.825126 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.825112    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.825260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.825243    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.932313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.930769    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mws\" (UniqueName: \"kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.932313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.930900    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.932313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.931008    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.932313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.931174    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.932724 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.932405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.948280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.945332    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.951239 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.951200    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:43.955897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:43.955862    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mws\" (UniqueName: \"kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws\") pod \"maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:44.055139 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:44.055042    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:44.415360 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:00:44.415325    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c686a0b_589f_4436_a395_ad20a393e79c.slice/crio-5197d07761dd586d9af6997c2e603a971fd85291931e25a4a003157000f741c3 WatchSource:0}: Error finding container 5197d07761dd586d9af6997c2e603a971fd85291931e25a4a003157000f741c3: Status 404 returned error can't find the container with id 5197d07761dd586d9af6997c2e603a971fd85291931e25a4a003157000f741c3
Oct 31 20:00:44.418392 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:44.418362    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"]
Oct 31 20:00:45.073287 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:45.073247    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r" event={"ID":"2c686a0b-589f-4436-a395-ad20a393e79c","Type":"ContainerStarted","Data":"52dfac1cecb70f5082015d89209ee24b56a6450be8710a7133a18377b90ff471"}
Oct 31 20:00:45.073734 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:45.073321    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:45.073734 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:45.073345    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r" event={"ID":"2c686a0b-589f-4436-a395-ad20a393e79c","Type":"ContainerStarted","Data":"5197d07761dd586d9af6997c2e603a971fd85291931e25a4a003157000f741c3"}
Oct 31 20:00:45.103299 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:45.103236    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r" podStartSLOduration=1.610700732 podStartE2EDuration="2.10321625s" totalImagesPullingTime="492.515518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:45.098129131 +0000 UTC m=+2438.559503744" watchObservedRunningTime="2026-10-31 20:00:45.10321625 +0000 UTC m=+2438.564590866"
Oct 31 20:00:47.050370 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:47.050275    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:00:50.198032 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:50.197993    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"]
Oct 31 20:00:50.198633 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:50.198217    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy" containerID="cri-o://5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3" gracePeriod=30
Oct 31 20:00:51.139294 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:51.139265    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:00:55.582203 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.582172    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 20:00:55.653842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.653808    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"]
Oct 31 20:00:55.660484 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660451    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.660662 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660514    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.660722 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660667    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.660815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660724    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.660815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660807    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.660943 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660857    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.661001 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.660976    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.661101 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.661082    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.661255 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.661178    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5\") pod \"708a5ebd-a589-4796-84d3-26d27841278d\" (UID: \"708a5ebd-a589-4796-84d3-26d27841278d\") "
Oct 31 20:00:55.661524 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.661497    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.661859 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.661838    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:55.662443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.662413    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/708a5ebd-a589-4796-84d3-26d27841278d-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.662607 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.662448    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.663713 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.663180    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.663713 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.663401    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.663713 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.663646    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.663713 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.663693    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.664140 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.664118    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:00:55.666026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.666004    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "kube-api-access-dd2r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:55.667678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.667647    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token" pod "708a5ebd-a589-4796-84d3-26d27841278d" (UID: "708a5ebd-a589-4796-84d3-26d27841278d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:55.764272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764236    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/708a5ebd-a589-4796-84d3-26d27841278d-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.764272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764264    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.764272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764274    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.764272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764283    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.764565 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764293    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.764565 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764303    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd2r5\" (UniqueName: \"kubernetes.io/projected/708a5ebd-a589-4796-84d3-26d27841278d-kube-api-access-dd2r5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.764565 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:55.764311    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/708a5ebd-a589-4796-84d3-26d27841278d-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:56.261556 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.261488    5958 generic.go:356] "Generic (PLEG): container finished" podID="708a5ebd-a589-4796-84d3-26d27841278d" containerID="5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3" exitCode=0
Oct 31 20:00:56.261737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.261565    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"
Oct 31 20:00:56.261737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.261577    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" event={"ID":"708a5ebd-a589-4796-84d3-26d27841278d","Type":"ContainerDied","Data":"5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3"}
Oct 31 20:00:56.261737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.261704    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv" event={"ID":"708a5ebd-a589-4796-84d3-26d27841278d","Type":"ContainerDied","Data":"06cd2f8ca7bca5633742c32ac30d0f2e2d4324ad65cbadb1dfd8cfdf729c33d9"}
Oct 31 20:00:56.261925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.261751    5958 scope.go:122] "RemoveContainer" containerID="5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3"
Oct 31 20:00:56.276811 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.276790    5958 scope.go:122] "RemoveContainer" containerID="5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3"
Oct 31 20:00:56.277158 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:00:56.277125    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3\": container with ID starting with 5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3 not found: ID does not exist" containerID="5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3"
Oct 31 20:00:56.277236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.277166    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3"} err="failed to get container status \"5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3\": rpc error: code = NotFound desc = could not find container \"5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3\": container with ID starting with 5e6bd6b6602274c9d6ad63d817b4abcf04aca4d6a2f8301a08457746f93498f3 not found: ID does not exist"
Oct 31 20:00:56.295508 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.295474    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"]
Oct 31 20:00:56.299744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.299718    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-pspcv"]
Oct 31 20:00:56.666696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:00:56.666667    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708a5ebd-a589-4796-84d3-26d27841278d" path="/var/lib/kubelet/pods/708a5ebd-a589-4796-84d3-26d27841278d/volumes"
Oct 31 20:01:01.163809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:01.163770    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"]
Oct 31 20:01:01.170828 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:01.170772    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"]
Oct 31 20:01:06.165264 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:06.165202    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerName="payload-processing" containerID="cri-o://43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6" gracePeriod=25
Oct 31 20:01:06.171866 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:06.171833    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" containerName="payload-pre-processing" containerID="cri-o://58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835" gracePeriod=25
Oct 31 20:01:10.180062 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:10.180027    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"]
Oct 31 20:01:10.409600 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:10.409525    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:01:10.698036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:10.697995    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main" containerID="cri-o://dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e" gracePeriod=45
Oct 31 20:01:11.055494 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.055463    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 20:01:11.235464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.235430    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs\") pod \"9627c756-873f-4326-90a0-190da4caf84a\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") "
Oct 31 20:01:11.235954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.235512    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v\" (UniqueName: \"kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v\") pod \"9627c756-873f-4326-90a0-190da4caf84a\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") "
Oct 31 20:01:11.235954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.235696    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home\") pod \"9627c756-873f-4326-90a0-190da4caf84a\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") "
Oct 31 20:01:11.235954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.235725    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache\") pod \"9627c756-873f-4326-90a0-190da4caf84a\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") "
Oct 31 20:01:11.235954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.235842    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm\") pod \"9627c756-873f-4326-90a0-190da4caf84a\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") "
Oct 31 20:01:11.235954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.235873    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir\") pod \"9627c756-873f-4326-90a0-190da4caf84a\" (UID: \"9627c756-873f-4326-90a0-190da4caf84a\") "
Oct 31 20:01:11.236647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.236626    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir" pod "9627c756-873f-4326-90a0-190da4caf84a" (UID: "9627c756-873f-4326-90a0-190da4caf84a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.237155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.237102    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home" pod "9627c756-873f-4326-90a0-190da4caf84a" (UID: "9627c756-873f-4326-90a0-190da4caf84a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.237557 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.237514    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache" pod "9627c756-873f-4326-90a0-190da4caf84a" (UID: "9627c756-873f-4326-90a0-190da4caf84a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.240084 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.239696    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs" pod "9627c756-873f-4326-90a0-190da4caf84a" (UID: "9627c756-873f-4326-90a0-190da4caf84a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:11.241346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.241080    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm" pod "9627c756-873f-4326-90a0-190da4caf84a" (UID: "9627c756-873f-4326-90a0-190da4caf84a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.241346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.241099    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v" pod "9627c756-873f-4326-90a0-190da4caf84a" (UID: "9627c756-873f-4326-90a0-190da4caf84a"). InnerVolumeSpecName "kube-api-access-vcs8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:11.338340 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.338305    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9627c756-873f-4326-90a0-190da4caf84a-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.338340 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.338339    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcs8v\" (UniqueName: \"kubernetes.io/projected/9627c756-873f-4326-90a0-190da4caf84a-kube-api-access-vcs8v\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.338577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.338349    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.338577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.338366    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.338577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.338375    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.338577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.338383    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9627c756-873f-4326-90a0-190da4caf84a-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.419489 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.419448    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:01:11.678190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.678158    5958 generic.go:356] "Generic (PLEG): container finished" podID="9627c756-873f-4326-90a0-190da4caf84a" containerID="dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e" exitCode=0
Oct 31 20:01:11.678373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.678204    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"
Oct 31 20:01:11.678373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.678236    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" event={"ID":"9627c756-873f-4326-90a0-190da4caf84a","Type":"ContainerDied","Data":"dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e"}
Oct 31 20:01:11.678373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.678283    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc" event={"ID":"9627c756-873f-4326-90a0-190da4caf84a","Type":"ContainerDied","Data":"9a6aa5d1012d0fcf80c3da75b3eff326ca16ab03398c666c451c91e7ab201633"}
Oct 31 20:01:11.678373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.678301    5958 scope.go:122] "RemoveContainer" containerID="dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e"
Oct 31 20:01:11.691068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.691047    5958 scope.go:122] "RemoveContainer" containerID="dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e"
Oct 31 20:01:11.691377 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:01:11.691355    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e\": container with ID starting with dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e not found: ID does not exist" containerID="dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e"
Oct 31 20:01:11.691424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.691385    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e"} err="failed to get container status \"dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e\": rpc error: code = NotFound desc = could not find container \"dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e\": container with ID starting with dedc49930fdc91a11cc79f5bdf9c99b7399b990a2c1025954001e81685a8f05e not found: ID does not exist"
Oct 31 20:01:11.715646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.715610    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"]
Oct 31 20:01:11.720570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:11.720528    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-2-w5-5bbf75-kserve-7bf5c66dd-56gqc"]
Oct 31 20:01:12.667957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:12.667915    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9627c756-873f-4326-90a0-190da4caf84a" path="/var/lib/kubelet/pods/9627c756-873f-4326-90a0-190da4caf84a/volumes"
Oct 31 20:01:15.180424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:15.180374    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx" podUID="28dc036c-2b1c-4f95-8929-b3e0f58c2300" containerName="payload-pre-processing" containerID="cri-o://7a873794beec0bc0f8586b4cc36d6f8f2b0d936c5a4245f198094122d70245c4" gracePeriod=25
Oct 31 20:01:15.897579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:15.892793    5958 generic.go:356] "Generic (PLEG): container finished" podID="28dc036c-2b1c-4f95-8929-b3e0f58c2300" containerID="7a873794beec0bc0f8586b4cc36d6f8f2b0d936c5a4245f198094122d70245c4" exitCode=0
Oct 31 20:01:15.897579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:15.896808    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx" event={"ID":"28dc036c-2b1c-4f95-8929-b3e0f58c2300","Type":"ContainerDied","Data":"7a873794beec0bc0f8586b4cc36d6f8f2b0d936c5a4245f198094122d70245c4"}
Oct 31 20:01:16.027006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.021458    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:01:16.194252 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.194218    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume\") pod \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") "
Oct 31 20:01:16.194778 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.194284    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m\" (UniqueName: \"kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m\") pod \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\" (UID: \"28dc036c-2b1c-4f95-8929-b3e0f58c2300\") "
Oct 31 20:01:16.194778 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.194745    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume" pod "28dc036c-2b1c-4f95-8929-b3e0f58c2300" (UID: "28dc036c-2b1c-4f95-8929-b3e0f58c2300"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:16.194992 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.194972    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc036c-2b1c-4f95-8929-b3e0f58c2300-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:16.196522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.196493    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m" pod "28dc036c-2b1c-4f95-8929-b3e0f58c2300" (UID: "28dc036c-2b1c-4f95-8929-b3e0f58c2300"). InnerVolumeSpecName "kube-api-access-q5b4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:16.384568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.384273    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5b4m\" (UniqueName: \"kubernetes.io/projected/28dc036c-2b1c-4f95-8929-b3e0f58c2300-kube-api-access-q5b4m\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:16.931566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.930084    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx" event={"ID":"28dc036c-2b1c-4f95-8929-b3e0f58c2300","Type":"ContainerDied","Data":"2ae2212d677efc5833bdacd417041a17a20e14d66c82b03204cbba802e212b02"}
Oct 31 20:01:16.931566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.930170    5958 scope.go:122] "RemoveContainer" containerID="7a873794beec0bc0f8586b4cc36d6f8f2b0d936c5a4245f198094122d70245c4"
Oct 31 20:01:16.931566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.930346    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"
Oct 31 20:01:16.981946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.981909    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"]
Oct 31 20:01:16.990766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:16.990258    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-c79bb986c-nxcxx"]
Oct 31 20:01:17.054720 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:01:17.054676    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:01:18.668842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:18.668811    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dc036c-2b1c-4f95-8929-b3e0f58c2300" path="/var/lib/kubelet/pods/28dc036c-2b1c-4f95-8929-b3e0f58c2300/volumes"
Oct 31 20:01:20.416143 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:20.416101    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:01:21.420072 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:21.420027    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:01:24.435698 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.435659    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"]
Oct 31 20:01:24.436478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.436420    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk" podUID="63ee7911-d643-47bd-bd64-031db7fdd5ae" containerName="maas-api" containerID="cri-o://7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5" gracePeriod=30
Oct 31 20:01:24.801564 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.798960    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 20:01:24.916815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.915138    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls\") pod \"63ee7911-d643-47bd-bd64-031db7fdd5ae\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") "
Oct 31 20:01:24.916815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.915290    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f\" (UniqueName: \"kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f\") pod \"63ee7911-d643-47bd-bd64-031db7fdd5ae\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") "
Oct 31 20:01:24.916815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.915324    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca\") pod \"63ee7911-d643-47bd-bd64-031db7fdd5ae\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") "
Oct 31 20:01:24.916815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.915483    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs\") pod \"63ee7911-d643-47bd-bd64-031db7fdd5ae\" (UID: \"63ee7911-d643-47bd-bd64-031db7fdd5ae\") "
Oct 31 20:01:24.917510 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.917463    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca" pod "63ee7911-d643-47bd-bd64-031db7fdd5ae" (UID: "63ee7911-d643-47bd-bd64-031db7fdd5ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:24.920486 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.920451    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs" pod "63ee7911-d643-47bd-bd64-031db7fdd5ae" (UID: "63ee7911-d643-47bd-bd64-031db7fdd5ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:24.920664 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.920502    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f" pod "63ee7911-d643-47bd-bd64-031db7fdd5ae" (UID: "63ee7911-d643-47bd-bd64-031db7fdd5ae"). InnerVolumeSpecName "kube-api-access-mmt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:24.925601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:24.923661    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls" pod "63ee7911-d643-47bd-bd64-031db7fdd5ae" (UID: "63ee7911-d643-47bd-bd64-031db7fdd5ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:25.027490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.027364    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.028287 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.027853    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmt8f\" (UniqueName: \"kubernetes.io/projected/63ee7911-d643-47bd-bd64-031db7fdd5ae-kube-api-access-mmt8f\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.028785 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.028762    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ee7911-d643-47bd-bd64-031db7fdd5ae-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.028954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.028941    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63ee7911-d643-47bd-bd64-031db7fdd5ae-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.054274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.054231    5958 generic.go:356] "Generic (PLEG): container finished" podID="63ee7911-d643-47bd-bd64-031db7fdd5ae" containerID="7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5" exitCode=0
Oct 31 20:01:25.056185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.054674    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"
Oct 31 20:01:25.056185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.054922    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk" event={"ID":"63ee7911-d643-47bd-bd64-031db7fdd5ae","Type":"ContainerDied","Data":"7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5"}
Oct 31 20:01:25.056185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.054979    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk" event={"ID":"63ee7911-d643-47bd-bd64-031db7fdd5ae","Type":"ContainerDied","Data":"844dabb01e919c87dd8b86e02c8703c5edef45decec732fd5ca81ef3a0d2fabb"}
Oct 31 20:01:25.056185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.055009    5958 scope.go:122] "RemoveContainer" containerID="7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5"
Oct 31 20:01:25.078463 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.078390    5958 scope.go:122] "RemoveContainer" containerID="7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5"
Oct 31 20:01:25.079168 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:01:25.079076    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5\": container with ID starting with 7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5 not found: ID does not exist" containerID="7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5"
Oct 31 20:01:25.079168 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.079119    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5"} err="failed to get container status \"7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5\": rpc error: code = NotFound desc = could not find container \"7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5\": container with ID starting with 7b8412b7708281d943f6e14bcd5fa8fab7f110f3c18a7d7021c5b3cf2c66dde5 not found: ID does not exist"
Oct 31 20:01:25.093007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.092965    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"]
Oct 31 20:01:25.097582 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:25.097533    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-26cd59-6b9b7c6f69-2d9gk"]
Oct 31 20:01:26.523005 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.522977    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 20:01:26.526679 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.526654    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 20:01:26.573699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.572516    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch\" (UniqueName: \"kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch\") pod \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") "
Oct 31 20:01:26.573699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.572705    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume\") pod \"b977c030-04a9-4062-a654-ddbd443b3efd\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") "
Oct 31 20:01:26.573699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.572759    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf\" (UniqueName: \"kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf\") pod \"b977c030-04a9-4062-a654-ddbd443b3efd\" (UID: \"b977c030-04a9-4062-a654-ddbd443b3efd\") "
Oct 31 20:01:26.573699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.572905    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume\") pod \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\" (UID: \"1c24cbec-c972-4d27-8ff5-0edcd14b20e1\") "
Oct 31 20:01:26.574496 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.574451    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume" pod "1c24cbec-c972-4d27-8ff5-0edcd14b20e1" (UID: "1c24cbec-c972-4d27-8ff5-0edcd14b20e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:26.579862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.579832    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf" pod "b977c030-04a9-4062-a654-ddbd443b3efd" (UID: "b977c030-04a9-4062-a654-ddbd443b3efd"). InnerVolumeSpecName "kube-api-access-77jsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:26.580058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.580035    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume" pod "b977c030-04a9-4062-a654-ddbd443b3efd" (UID: "b977c030-04a9-4062-a654-ddbd443b3efd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:26.583405 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.583384    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch" pod "1c24cbec-c972-4d27-8ff5-0edcd14b20e1" (UID: "1c24cbec-c972-4d27-8ff5-0edcd14b20e1"). InnerVolumeSpecName "kube-api-access-rnkch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:26.668370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.668338    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ee7911-d643-47bd-bd64-031db7fdd5ae" path="/var/lib/kubelet/pods/63ee7911-d643-47bd-bd64-031db7fdd5ae/volumes"
Oct 31 20:01:26.675572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.675527    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b977c030-04a9-4062-a654-ddbd443b3efd-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:26.675572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.675574    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jsf\" (UniqueName: \"kubernetes.io/projected/b977c030-04a9-4062-a654-ddbd443b3efd-kube-api-access-77jsf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:26.675758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.675591    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:26.675758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:26.675606    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnkch\" (UniqueName: \"kubernetes.io/projected/1c24cbec-c972-4d27-8ff5-0edcd14b20e1-kube-api-access-rnkch\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:27.077034 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.077004    5958 generic.go:356] "Generic (PLEG): container finished" podID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerID="43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6" exitCode=0
Oct 31 20:01:27.077198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.077060    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"
Oct 31 20:01:27.077198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.077094    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" event={"ID":"1c24cbec-c972-4d27-8ff5-0edcd14b20e1","Type":"ContainerDied","Data":"43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6"}
Oct 31 20:01:27.077198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.077158    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4" event={"ID":"1c24cbec-c972-4d27-8ff5-0edcd14b20e1","Type":"ContainerDied","Data":"2334779abd4fc87433f7e6e2b41c067726631ca85bb34ed66dcc601ac37ecea5"}
Oct 31 20:01:27.077198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.077192    5958 scope.go:122] "RemoveContainer" containerID="43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6"
Oct 31 20:01:27.079518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.079413    5958 generic.go:356] "Generic (PLEG): container finished" podID="b977c030-04a9-4062-a654-ddbd443b3efd" containerID="58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835" exitCode=0
Oct 31 20:01:27.079518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.079451    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" event={"ID":"b977c030-04a9-4062-a654-ddbd443b3efd","Type":"ContainerDied","Data":"58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835"}
Oct 31 20:01:27.079518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.079491    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"
Oct 31 20:01:27.079518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.079512    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw" event={"ID":"b977c030-04a9-4062-a654-ddbd443b3efd","Type":"ContainerDied","Data":"f1a7fe07d075bcb9f02e192997778782bd2ab05d178799fe6e9efc11a44d2af2"}
Oct 31 20:01:27.091194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.091101    5958 scope.go:122] "RemoveContainer" containerID="43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6"
Oct 31 20:01:27.092580 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:01:27.092520    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6\": container with ID starting with 43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6 not found: ID does not exist" containerID="43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6"
Oct 31 20:01:27.092686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.092571    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6"} err="failed to get container status \"43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6\": rpc error: code = NotFound desc = could not find container \"43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6\": container with ID starting with 43e511751cae22ca1dbdd67608a988091e5644eb18f7652afd0fc503748901b6 not found: ID does not exist"
Oct 31 20:01:27.092686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.092593    5958 scope.go:122] "RemoveContainer" containerID="58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835"
Oct 31 20:01:27.104968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.104921    5958 scope.go:122] "RemoveContainer" containerID="58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835"
Oct 31 20:01:27.105655 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:01:27.105272    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835\": container with ID starting with 58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835 not found: ID does not exist" containerID="58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835"
Oct 31 20:01:27.105655 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.105309    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835"} err="failed to get container status \"58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835\": rpc error: code = NotFound desc = could not find container \"58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835\": container with ID starting with 58829bd6cd3a962ff6842c393a2a9b6fa2c5579c2efa86daa64a4c4894967835 not found: ID does not exist"
Oct 31 20:01:27.189656 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.189614    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"]
Oct 31 20:01:27.237668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.237626    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5bbf75-6db9b4cfd7-j4pv4"]
Oct 31 20:01:27.268207 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.268165    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"]
Oct 31 20:01:27.271642 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:27.271608    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bbf75-7fc4ff59f9-678zw"]
Oct 31 20:01:28.668701 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:28.668654    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" path="/var/lib/kubelet/pods/1c24cbec-c972-4d27-8ff5-0edcd14b20e1/volumes"
Oct 31 20:01:28.670399 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:28.670371    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" path="/var/lib/kubelet/pods/b977c030-04a9-4062-a654-ddbd443b3efd/volumes"
Oct 31 20:01:45.029353 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.029314    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"]
Oct 31 20:01:45.030490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030465    5958 cpu_manager.go:414] "containerMap: removing container" podUID="28dc036c-2b1c-4f95-8929-b3e0f58c2300" containerName="payload-pre-processing"
Oct 31 20:01:45.030490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030488    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dc036c-2b1c-4f95-8929-b3e0f58c2300" containerName="payload-pre-processing"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030525    5958 cpu_manager.go:414] "containerMap: removing container" podUID="63ee7911-d643-47bd-bd64-031db7fdd5ae" containerName="maas-api"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030531    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ee7911-d643-47bd-bd64-031db7fdd5ae" containerName="maas-api"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030571    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030577    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030599    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerName="payload-processing"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030604    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerName="payload-processing"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030616    5958 cpu_manager.go:414] "containerMap: removing container" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030621    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030630    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" containerName="payload-pre-processing"
Oct 31 20:01:45.030683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030635    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" containerName="payload-pre-processing"
Oct 31 20:01:45.030994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030757    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c24cbec-c972-4d27-8ff5-0edcd14b20e1" containerName="payload-processing"
Oct 31 20:01:45.030994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030769    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="708a5ebd-a589-4796-84d3-26d27841278d" containerName="istio-proxy"
Oct 31 20:01:45.030994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030775    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b977c030-04a9-4062-a654-ddbd443b3efd" containerName="payload-pre-processing"
Oct 31 20:01:45.030994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030782    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dc036c-2b1c-4f95-8929-b3e0f58c2300" containerName="payload-pre-processing"
Oct 31 20:01:45.030994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030795    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ee7911-d643-47bd-bd64-031db7fdd5ae" containerName="maas-api"
Oct 31 20:01:45.030994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.030805    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9627c756-873f-4326-90a0-190da4caf84a" containerName="main"
Oct 31 20:01:45.035318 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.035299    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.101254 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.101220    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"]
Oct 31 20:01:45.101885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.101852    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.102089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.101944    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.102275 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.102203    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.102486 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.102463    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.102717 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.102691    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.102894 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.102758    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.102894 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.102825    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.103776 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.103072    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8sj\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.103776 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.103129    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.137726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.137695    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c734f5-openshift-default-dockercfg-jg24l\""
Oct 31 20:01:45.204236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204197    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204248    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204303    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204352    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204577    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8sj\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204706    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204725    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.204980 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.204962    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.205073 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.205054    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.205151 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.205128    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.205208 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.205142    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.205208 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.205195    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.205323 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.205243    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.205386 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.205358    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.207242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.207220    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.207330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.207251    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.236476 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.236441    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.242280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.242246    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8sj\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.353185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.353097    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:45.671458 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:01:45.671424    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1880b0c3_b0c0_40f4_b21e_3885303d14da.slice/crio-e5b2c958e7c0401a87cf05d36837c62c9400c80d4e5e52d59381b745de2d662f WatchSource:0}: Error finding container e5b2c958e7c0401a87cf05d36837c62c9400c80d4e5e52d59381b745de2d662f: Status 404 returned error can't find the container with id e5b2c958e7c0401a87cf05d36837c62c9400c80d4e5e52d59381b745de2d662f
Oct 31 20:01:45.672214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.672185    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"]
Oct 31 20:01:45.674590 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.674536    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:01:45.674691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.674623    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:01:45.674691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:45.674657    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:01:46.431704 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:46.431665    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" event={"ID":"1880b0c3-b0c0-40f4-b21e-3885303d14da","Type":"ContainerStarted","Data":"3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d"}
Oct 31 20:01:46.432164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:46.431721    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" event={"ID":"1880b0c3-b0c0-40f4-b21e-3885303d14da","Type":"ContainerStarted","Data":"e5b2c958e7c0401a87cf05d36837c62c9400c80d4e5e52d59381b745de2d662f"}
Oct 31 20:01:46.469036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:46.468980    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podStartSLOduration=1.468964699 podStartE2EDuration="1.468964699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:46.462161024 +0000 UTC m=+2499.923535641" watchObservedRunningTime="2026-10-31 20:01:46.468964699 +0000 UTC m=+2499.930339313"
Oct 31 20:01:47.061680 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:01:47.061643    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:01:47.354006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:47.353875    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:47.355468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:47.355436    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 20:01:47.355625 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:47.355525    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 20:01:48.354623 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:48.354571    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 20:01:48.355255 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:48.354654    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 20:01:49.353794 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.353754    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 20:01:49.353988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.353833    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 20:01:49.553875 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.553837    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"]
Oct 31 20:01:49.558893 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.558867    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.570247 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.568561    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e1d060-openshift-default-dockercfg-w5qst\""
Oct 31 20:01:49.575749 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.575721    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"]
Oct 31 20:01:49.636340 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636238    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.636592 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636405    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t579d\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.636682 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636593    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.636738 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636696    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.636829 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636811    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.636910 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636885    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.640623 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.636959    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.640623 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.637032    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.640623 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.637061    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.738581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.738499    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.738581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.738558    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.738865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.738785    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.738996    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739014    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739124 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739061    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t579d\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739243 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739228    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739306    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739418 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739398    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739607 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739442    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739607 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739489    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739856 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739834    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.739937 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.739913    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.740187 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.740171    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.741631 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.741609    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.741726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.741706    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.752140 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.752109    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.752274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.752151    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t579d\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:49.878226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:49.878195    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:50.354140 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.354059    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 20:01:50.354140 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.354111    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 20:01:50.402321 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:01:50.402286    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847c2db_ec3f_4e41_8140_5e7a9db18309.slice/crio-1edce90f748c835bc05850076691e7cacd6ceb56eac08c78259f0e7e04cce303 WatchSource:0}: Error finding container 1edce90f748c835bc05850076691e7cacd6ceb56eac08c78259f0e7e04cce303: Status 404 returned error can't find the container with id 1edce90f748c835bc05850076691e7cacd6ceb56eac08c78259f0e7e04cce303
Oct 31 20:01:50.422567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.403512    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"]
Oct 31 20:01:50.432284 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.432240    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:01:50.432433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.432335    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:01:50.432433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.432401    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:01:50.547353 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.547317    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" event={"ID":"a847c2db-ec3f-4e41-8140-5e7a9db18309","Type":"ContainerStarted","Data":"6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478"}
Oct 31 20:01:50.547585 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.547374    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" event={"ID":"a847c2db-ec3f-4e41-8140-5e7a9db18309","Type":"ContainerStarted","Data":"1edce90f748c835bc05850076691e7cacd6ceb56eac08c78259f0e7e04cce303"}
Oct 31 20:01:50.577586 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.577484    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podStartSLOduration=1.577466391 podStartE2EDuration="1.577466391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:50.574015805 +0000 UTC m=+2504.035390421" watchObservedRunningTime="2026-10-31 20:01:50.577466391 +0000 UTC m=+2504.038841082"
Oct 31 20:01:50.878992 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:50.878957    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:51.354393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:51.354302    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 31 20:01:51.354393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:51.354363    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 31 20:01:52.029568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.028932    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 20:01:52.029568 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.029001    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 20:01:52.462290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.462252    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:52.463840 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.463813    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:52.464172 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.464147    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"]
Oct 31 20:01:52.464570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.464502    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" containerID="cri-o://d95998a1a3b330c0a06c448a9735af0196546d036be6a4ef20f3c39bfbc7d7ba" gracePeriod=30
Oct 31 20:01:52.467175 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.467151    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:01:52.881464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.881003    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 20:01:52.881464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:52.881079    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 20:01:53.878531 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:53.878496    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 20:01:53.879107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:53.878593    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 20:01:54.878538 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:54.878494    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 20:01:54.879009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:54.878615    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 20:01:55.878598 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:55.878501    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 20:01:55.879036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:55.878619    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 20:01:56.878608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:56.878567    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 31 20:01:56.879142 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:56.878648    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 31 20:01:57.435660 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.434116    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:01:57.435660 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.434206    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:01:57.758698 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.758300    5958 generic.go:356] "Generic (PLEG): container finished" podID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerID="d95998a1a3b330c0a06c448a9735af0196546d036be6a4ef20f3c39bfbc7d7ba" exitCode=0
Oct 31 20:01:57.758698 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.758483    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" event={"ID":"b39a418e-9e97-41cb-8753-5fe2c2f8716b","Type":"ContainerDied","Data":"d95998a1a3b330c0a06c448a9735af0196546d036be6a4ef20f3c39bfbc7d7ba"}
Oct 31 20:01:57.835577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.835528    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 20:01:57.980471 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.980435    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:57.980964 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.980797    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:57.981790 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.981770    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:01:57.997705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.997672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.997905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.997750    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.997905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.997815    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.997905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.997870    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.998099 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.997918    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.998099 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998000    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.998099 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998036    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.998099 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998060    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:57.998319 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998100    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.998319 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998206    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo\") pod \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\" (UID: \"b39a418e-9e97-41cb-8753-5fe2c2f8716b\") "
Oct 31 20:01:57.998428 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998316    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:57.998428 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998318    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:57.998589 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.998533    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:57.999153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.999075    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:57.999153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.999101    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:57.999153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.999101    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:57.999153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.999116    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:57.999153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:57.999130    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:58.000242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.000218    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "kube-api-access-dvmxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:58.000373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.000354    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:58.000756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.000738    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:01:58.000859 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.000843    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy" pod "b39a418e-9e97-41cb-8753-5fe2c2f8716b" (UID: "b39a418e-9e97-41cb-8753-5fe2c2f8716b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:58.101141 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.101105    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:58.101352 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.101329    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvmxf\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-kube-api-access-dvmxf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:58.101352 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.101351    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:58.101874 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.101361    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:58.101874 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.101369    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b39a418e-9e97-41cb-8753-5fe2c2f8716b-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:58.428187 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.428142    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"]
Oct 31 20:01:58.429138 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.429116    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy"
Oct 31 20:01:58.429138 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.429138    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy"
Oct 31 20:01:58.429311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.429276    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" containerName="istio-proxy"
Oct 31 20:01:58.434801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.434775    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.442353 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.442315    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"]
Oct 31 20:01:58.444697 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.444675    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d7647c\""
Oct 31 20:01:58.444835 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.444724    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d7647c-dockercfg-ql9mf\""
Oct 31 20:01:58.455227 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.455192    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"]
Oct 31 20:01:58.460960 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.460935    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.481990 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.479568    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"]
Oct 31 20:01:58.617933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.609873    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.617933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.610147    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.617933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.610317    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbxn\" (UniqueName: \"kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.617933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.610416    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgj4x\" (UniqueName: \"kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x\") pod \"payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.728625 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.717808    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgj4x\" (UniqueName: \"kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x\") pod \"payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.728625 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.717961    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.728625 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.718246    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.728625 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.718415    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbxn\" (UniqueName: \"kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.728625 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.720102    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.739583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.735930    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.761567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.749521    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbxn\" (UniqueName: \"kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.794957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.786892    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgj4x\" (UniqueName: \"kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x\") pod \"payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:58.794957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.788533    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc" event={"ID":"b39a418e-9e97-41cb-8753-5fe2c2f8716b","Type":"ContainerDied","Data":"234b85f78d475020e6a4f48bb51b2ef3c9752e62f74789cf1fdd3d28f034e8b7"}
Oct 31 20:01:58.794957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.788625    5958 scope.go:122] "RemoveContainer" containerID="d95998a1a3b330c0a06c448a9735af0196546d036be6a4ef20f3c39bfbc7d7ba"
Oct 31 20:01:58.794957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.790980    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:58.794957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.791179    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"
Oct 31 20:01:58.859177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.859138    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"]
Oct 31 20:01:58.864528 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:58.864467    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-9m9rc"]
Oct 31 20:01:59.059438 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.059349    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:59.134936 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.134895    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"]
Oct 31 20:01:59.144166 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:01:59.144093    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda795c033_7053_445a_b2c9_b6664c756aae.slice/crio-70b8215da3985bf489494bb8343d45e113e2a208fe69b99f4da9e98ebbbe7820 WatchSource:0}: Error finding container 70b8215da3985bf489494bb8343d45e113e2a208fe69b99f4da9e98ebbbe7820: Status 404 returned error can't find the container with id 70b8215da3985bf489494bb8343d45e113e2a208fe69b99f4da9e98ebbbe7820
Oct 31 20:01:59.351433 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:01:59.351394    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c5d5b3_ab78_4dbe_ae7d_b1bfe489e5e0.slice/crio-405ed6c2dadf86e801870bf4c2e2a53197d7dcd149fc5bdc2249b9e8f845b976 WatchSource:0}: Error finding container 405ed6c2dadf86e801870bf4c2e2a53197d7dcd149fc5bdc2249b9e8f845b976: Status 404 returned error can't find the container with id 405ed6c2dadf86e801870bf4c2e2a53197d7dcd149fc5bdc2249b9e8f845b976
Oct 31 20:01:59.351981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.351954    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"]
Oct 31 20:01:59.799113 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.799074    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc" event={"ID":"a795c033-7053-445a-b2c9-b6664c756aae","Type":"ContainerStarted","Data":"a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e"}
Oct 31 20:01:59.799325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.799130    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc" event={"ID":"a795c033-7053-445a-b2c9-b6664c756aae","Type":"ContainerStarted","Data":"70b8215da3985bf489494bb8343d45e113e2a208fe69b99f4da9e98ebbbe7820"}
Oct 31 20:01:59.799325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.799155    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:01:59.800794 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.800769    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw" event={"ID":"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0","Type":"ContainerStarted","Data":"04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b"}
Oct 31 20:01:59.800925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.800824    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:01:59.800925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.800851    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw" event={"ID":"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0","Type":"ContainerStarted","Data":"405ed6c2dadf86e801870bf4c2e2a53197d7dcd149fc5bdc2249b9e8f845b976"}
Oct 31 20:01:59.820554 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.820482    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc" podStartSLOduration=1.820467431 podStartE2EDuration="1.820467431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:59.819480321 +0000 UTC m=+2513.280854934" watchObservedRunningTime="2026-10-31 20:01:59.820467431 +0000 UTC m=+2513.281842098"
Oct 31 20:01:59.844716 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:01:59.844649    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw" podStartSLOduration=1.844629169 podStartE2EDuration="1.844629169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:59.842849363 +0000 UTC m=+2513.304223979" watchObservedRunningTime="2026-10-31 20:01:59.844629169 +0000 UTC m=+2513.306003787"
Oct 31 20:02:00.667221 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:00.667182    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39a418e-9e97-41cb-8753-5fe2c2f8716b" path="/var/lib/kubelet/pods/b39a418e-9e97-41cb-8753-5fe2c2f8716b/volumes"
Oct 31 20:02:05.255352 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:05.255314    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:02:07.264614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.264508    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"]
Oct 31 20:02:07.265248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.264770    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll" podUID="3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" containerName="maas-api" containerID="cri-o://ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd" gracePeriod=30
Oct 31 20:02:07.759309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.759273    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:02:07.817351 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.817317    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs\") pod \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") "
Oct 31 20:02:07.817588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.817506    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls\") pod \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") "
Oct 31 20:02:07.817588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.817537    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f\" (UniqueName: \"kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f\") pod \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") "
Oct 31 20:02:07.817731 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.817714    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca\") pod \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\" (UID: \"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6\") "
Oct 31 20:02:07.818447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.818071    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca" pod "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" (UID: "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:07.818447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.818421    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:07.821470 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.821434    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f" pod "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" (UID: "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6"). InnerVolumeSpecName "kube-api-access-8j82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:07.823966 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.823941    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls" pod "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" (UID: "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:07.825877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.825848    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs" pod "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" (UID: "3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:07.878217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.878175    5958 generic.go:356] "Generic (PLEG): container finished" podID="3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" containerID="ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd" exitCode=0
Oct 31 20:02:07.878505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.878482    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"
Oct 31 20:02:07.878628 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.878597    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll" event={"ID":"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6","Type":"ContainerDied","Data":"ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd"}
Oct 31 20:02:07.878755 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.878652    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll" event={"ID":"3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6","Type":"ContainerDied","Data":"98737962302f2352bf099da149bd3ae26bf1ff17d886f7f564e24f99de3ee331"}
Oct 31 20:02:07.878755 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.878678    5958 scope.go:122] "RemoveContainer" containerID="ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd"
Oct 31 20:02:07.893868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.893839    5958 scope.go:122] "RemoveContainer" containerID="ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd"
Oct 31 20:02:07.894204 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:07.894184    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd\": container with ID starting with ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd not found: ID does not exist" containerID="ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd"
Oct 31 20:02:07.894281 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.894214    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd"} err="failed to get container status \"ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd\": rpc error: code = NotFound desc = could not find container \"ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd\": container with ID starting with ecf409c58f2e78b5982fa19693438c61b6cf3bc42759456f58bfcd0772d057bd not found: ID does not exist"
Oct 31 20:02:07.912004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.911967    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"]
Oct 31 20:02:07.917108 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.917074    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5f3eff-7fc5d888c6-hzjll"]
Oct 31 20:02:07.920325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.920297    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:07.920325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.920326    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:07.920478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:07.920337    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j82f\" (UniqueName: \"kubernetes.io/projected/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6-kube-api-access-8j82f\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:08.669053 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:08.669019    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" path="/var/lib/kubelet/pods/3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6/volumes"
Oct 31 20:02:10.827591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:10.819266    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:02:10.827591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:10.822557    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:02:15.146896 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.146851    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"]
Oct 31 20:02:15.148285 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.148257    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" containerName="maas-api"
Oct 31 20:02:15.148285 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.148286    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" containerName="maas-api"
Oct 31 20:02:15.148506 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.148493    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb43a3d-0cd6-4ed5-9227-dfbc759be3d6" containerName="maas-api"
Oct 31 20:02:15.155005 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.154978    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.183110 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.183074    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"]
Oct 31 20:02:15.214640 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.214600    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.214878 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.214703    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.214878 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.214761    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.215012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.214990    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.215133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.215100    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.215216 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.215166    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.215281 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.215261    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkw4m\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.215361 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.215347    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.215436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.215421    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.316760 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.316718    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.316979 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.316775    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.316979 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.316818    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.316979 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.316854    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkw4m\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.316979 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.316902    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.316979 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.316967    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317097    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317155    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317230    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317275    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317379    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317594 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317515    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317666 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317648    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.317994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.317966    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.319236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.319215    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.319668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.319650    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.330842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.330814    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkw4m\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.330842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.330834    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.500320 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.500219    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:15.813342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.813306    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"]
Oct 31 20:02:15.813509 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:02:15.813312    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode374bf0e_a2e2_49aa_b9a4_b0bb77753193.slice/crio-cc5972dff86f87f0abc4ef58555827e5454ecb10178c26172cb8a3c7e561f9e1 WatchSource:0}: Error finding container cc5972dff86f87f0abc4ef58555827e5454ecb10178c26172cb8a3c7e561f9e1: Status 404 returned error can't find the container with id cc5972dff86f87f0abc4ef58555827e5454ecb10178c26172cb8a3c7e561f9e1
Oct 31 20:02:15.817448 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.817410    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:02:15.817575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.817492    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:02:15.817575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:15.817523    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:02:16.008772 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.008286    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" event={"ID":"e374bf0e-a2e2-49aa-b9a4-b0bb77753193","Type":"ContainerStarted","Data":"e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b"}
Oct 31 20:02:16.008772 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.008467    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" event={"ID":"e374bf0e-a2e2-49aa-b9a4-b0bb77753193","Type":"ContainerStarted","Data":"cc5972dff86f87f0abc4ef58555827e5454ecb10178c26172cb8a3c7e561f9e1"}
Oct 31 20:02:16.011429 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.011399    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"]
Oct 31 20:02:16.016803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.016769    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.025591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.023950    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"]
Oct 31 20:02:16.025591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.025093    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a84fcb\""
Oct 31 20:02:16.025591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.025273    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a84fcb-dockercfg-4qfs5\""
Oct 31 20:02:16.032253 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.032170    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.032253 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.032231    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzr5k\" (UniqueName: \"kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.056591 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.056506    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podStartSLOduration=1.056484599 podStartE2EDuration="1.056484599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:16.046722635 +0000 UTC m=+2529.508097254" watchObservedRunningTime="2026-10-31 20:02:16.056484599 +0000 UTC m=+2529.517859217"
Oct 31 20:02:16.120439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.120358    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"]
Oct 31 20:02:16.127289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.127263    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.134842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.134782    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.136089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.136066    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzr5k\" (UniqueName: \"kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.136583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.136402    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.140720 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.139623    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"]
Oct 31 20:02:16.157288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.157256    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzr5k\" (UniqueName: \"kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.391395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.391053    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:16.469209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.459681    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zt7\" (UniqueName: \"kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7\") pod \"payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.469209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.459849    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.516608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.504704    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:16.524738 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.524701    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:16.524987 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.524962    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:16.670958 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.662918    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zt7\" (UniqueName: \"kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7\") pod \"payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.705115 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.705076    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.710721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.710686    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.752452 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.752202    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zt7\" (UniqueName: \"kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7\") pod \"payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:16.771332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:16.770854    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:17.085530 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:17.085389    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:02:17.140906 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:17.140691    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"]
Oct 31 20:02:17.144903 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:02:17.144872    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b94853c_f3f6_4f54_994b_218aff398894.slice/crio-d561c5fe886bf87725bd6586bf050132d11fb5734e11f4a391adf9703516cd8f WatchSource:0}: Error finding container d561c5fe886bf87725bd6586bf050132d11fb5734e11f4a391adf9703516cd8f: Status 404 returned error can't find the container with id d561c5fe886bf87725bd6586bf050132d11fb5734e11f4a391adf9703516cd8f
Oct 31 20:02:17.327608 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:02:17.325121    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb54d47_d2ee_4b55_9f85_a8c83752cf1f.slice/crio-ddb01521173ea877eea55cbee72ec8431aed3c898d08b9e8bd7f8bede91377e2 WatchSource:0}: Error finding container ddb01521173ea877eea55cbee72ec8431aed3c898d08b9e8bd7f8bede91377e2: Status 404 returned error can't find the container with id ddb01521173ea877eea55cbee72ec8431aed3c898d08b9e8bd7f8bede91377e2
Oct 31 20:02:17.337233 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:17.337128    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"]
Oct 31 20:02:17.503033 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:17.502932    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:17.503033 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:17.502992    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:18.062512 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.062475    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f" event={"ID":"9b94853c-f3f6-4f54-994b-218aff398894","Type":"ContainerStarted","Data":"afeb9e1a65e3725b7fd26048f965795391117399fc9b1c1dfd8cccc8a9af3601"}
Oct 31 20:02:18.062754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.062537    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f" event={"ID":"9b94853c-f3f6-4f54-994b-218aff398894","Type":"ContainerStarted","Data":"d561c5fe886bf87725bd6586bf050132d11fb5734e11f4a391adf9703516cd8f"}
Oct 31 20:02:18.062754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.062608    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:18.064494 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.064464    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p" event={"ID":"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f","Type":"ContainerStarted","Data":"51612c14c7b4fee2f67de46700d3f8d2218be532fb95190ab6eff357991f004b"}
Oct 31 20:02:18.064687 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.064519    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p" event={"ID":"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f","Type":"ContainerStarted","Data":"ddb01521173ea877eea55cbee72ec8431aed3c898d08b9e8bd7f8bede91377e2"}
Oct 31 20:02:18.064786 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.064767    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:18.086647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.086596    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f" podStartSLOduration=3.08658184 podStartE2EDuration="3.08658184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:18.083746395 +0000 UTC m=+2531.545121013" watchObservedRunningTime="2026-10-31 20:02:18.08658184 +0000 UTC m=+2531.547956454"
Oct 31 20:02:18.136423 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.136364    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p" podStartSLOduration=2.136344122 podStartE2EDuration="2.136344122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:18.132046158 +0000 UTC m=+2531.593420772" watchObservedRunningTime="2026-10-31 20:02:18.136344122 +0000 UTC m=+2531.597718738"
Oct 31 20:02:18.503019 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.502984    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:18.503441 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:18.503038    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:19.502872 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:19.502836    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:19.503077 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:19.502896    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:20.502150 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:20.502111    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:20.502538 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:20.502476    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:21.502395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:21.502358    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:21.503009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:21.502976    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:22.510049 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:22.502669    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 31 20:02:22.510049 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:22.502744    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 31 20:02:23.507612 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:23.507539    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:23.508015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:23.507983    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:23.508803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:23.508782    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:02:27.138028 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.137892    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"]
Oct 31 20:02:27.150033 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.138273    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96" podUID="e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" containerName="maas-api" containerID="cri-o://fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126" gracePeriod=30
Oct 31 20:02:27.678120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.678090    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 20:02:27.758865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.758822    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs\") pod \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") "
Oct 31 20:02:27.759083 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.759011    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6\" (UniqueName: \"kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6\") pod \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") "
Oct 31 20:02:27.759209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.759186    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca\") pod \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") "
Oct 31 20:02:27.759348 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.759233    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls\") pod \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\" (UID: \"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9\") "
Oct 31 20:02:27.759705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.759678    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca" pod "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" (UID: "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:27.760373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.760349    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:27.762122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.762080    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs" pod "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" (UID: "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:27.762232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.762136    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6" pod "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" (UID: "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9"). InnerVolumeSpecName "kube-api-access-4cpf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:27.773945 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.773887    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls" pod "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" (UID: "e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:27.862809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.862772    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:27.862809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.862804    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:27.863095 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:27.862820    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cpf6\" (UniqueName: \"kubernetes.io/projected/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9-kube-api-access-4cpf6\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:28.216111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.216072    5958 generic.go:356] "Generic (PLEG): container finished" podID="e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" containerID="fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126" exitCode=0
Oct 31 20:02:28.216611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.216128    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"
Oct 31 20:02:28.216611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.216147    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96" event={"ID":"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9","Type":"ContainerDied","Data":"fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126"}
Oct 31 20:02:28.216611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.216220    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96" event={"ID":"e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9","Type":"ContainerDied","Data":"e909d9630964418f08b48819df683f4843e8ae48ff415ce48b6ef2b46fe5274c"}
Oct 31 20:02:28.216611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.216265    5958 scope.go:122] "RemoveContainer" containerID="fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126"
Oct 31 20:02:28.229683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.229652    5958 scope.go:122] "RemoveContainer" containerID="fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126"
Oct 31 20:02:28.230087 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:28.230048    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126\": container with ID starting with fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126 not found: ID does not exist" containerID="fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126"
Oct 31 20:02:28.230193 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.230087    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126"} err="failed to get container status \"fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126\": rpc error: code = NotFound desc = could not find container \"fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126\": container with ID starting with fcaef0de38829819ec2aee433873c4d7f2955ec754b1b9809c615972262d1126 not found: ID does not exist"
Oct 31 20:02:28.246680 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.246651    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"]
Oct 31 20:02:28.252214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.252184    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-fd1c91-656689f57b-q9r96"]
Oct 31 20:02:28.669462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:28.669424    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" path="/var/lib/kubelet/pods/e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9/volumes"
Oct 31 20:02:29.087455 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:29.087369    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:02:29.087455 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:29.087449    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:02:35.870523 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.870490    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"]
Oct 31 20:02:35.871478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.871460    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" containerName="maas-api"
Oct 31 20:02:35.871608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.871480    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" containerName="maas-api"
Oct 31 20:02:35.871860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.871843    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e51ec0-aac2-4216-8f04-7f0fc1bfd9d9" containerName="maas-api"
Oct 31 20:02:35.882569 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.877926    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:35.903864 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.903825    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a84fcb-serving-cert\""
Oct 31 20:02:35.908010 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.907973    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"]
Oct 31 20:02:35.951897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.951851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdpz\" (UniqueName: \"kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:35.952088 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.951922    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:35.952088 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.952044    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:35.952268 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:35.952241    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.061580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.057351    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdpz\" (UniqueName: \"kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.061580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.057438    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.061580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.057531    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.061580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.057671    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.061580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.059779    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.069007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.068946    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.070900 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.070872    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.078938 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.078901    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdpz\" (UniqueName: \"kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz\") pod \"maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.225071 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.224969    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:36.419242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.419169    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"]
Oct 31 20:02:36.419504 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.419475    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" containerID="cri-o://8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72" gracePeriod=30
Oct 31 20:02:36.612649 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:36.612607    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"]
Oct 31 20:02:37.321262 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:37.321219    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk" event={"ID":"607f4e4f-8aad-42d8-a162-f4235675ccac","Type":"ContainerStarted","Data":"0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97"}
Oct 31 20:02:37.321845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:37.321281    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk" event={"ID":"607f4e4f-8aad-42d8-a162-f4235675ccac","Type":"ContainerStarted","Data":"6317c54a586bd6dac98178338132277771e81180486a0655aaf955cc60562c3b"}
Oct 31 20:02:37.321845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:37.321331    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:37.343805 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:37.343749    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk" podStartSLOduration=1.8618498300000001 podStartE2EDuration="2.343735356s" totalImagesPullingTime="481.885526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:37.342779066 +0000 UTC m=+2550.804153681" watchObservedRunningTime="2026-10-31 20:02:37.343735356 +0000 UTC m=+2550.805109971"
Oct 31 20:02:40.531988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:40.531941    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:02:40.532464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:40.532012    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:02:42.172395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.172365    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 20:02:42.263522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263427    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.263522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263510    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.263824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263591    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.263824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263625    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.263824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263703    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.263824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263731    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.263824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263776    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:42.264111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.263984    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.264111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.264071    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.264224 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.264157    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket\") pod \"9d701ad4-cb46-46e5-94d4-15a01315912c\" (UID: \"9d701ad4-cb46-46e5-94d4-15a01315912c\") "
Oct 31 20:02:42.264920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.264482    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:42.264920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.264506    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:42.264920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.264579    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:42.264920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.264721    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:42.265218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.265161    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.265218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.265183    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.265218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.265200    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.265384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.265216    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.265384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.265234    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d701ad4-cb46-46e5-94d4-15a01315912c-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.266257 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.266226    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:42.266586 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.266527    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:02:42.267860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.267828    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:42.267968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.267901    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs" pod "9d701ad4-cb46-46e5-94d4-15a01315912c" (UID: "9d701ad4-cb46-46e5-94d4-15a01315912c"). InnerVolumeSpecName "kube-api-access-rdqcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:42.366756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.366725    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.366756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.366752    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.366756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.366763    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqcs\" (UniqueName: \"kubernetes.io/projected/9d701ad4-cb46-46e5-94d4-15a01315912c-kube-api-access-rdqcs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.367024 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.366774    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d701ad4-cb46-46e5-94d4-15a01315912c-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:42.371971 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.371936    5958 generic.go:356] "Generic (PLEG): container finished" podID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerID="8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72" exitCode=0
Oct 31 20:02:42.372171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.371982    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"
Oct 31 20:02:42.372171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.372022    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" event={"ID":"9d701ad4-cb46-46e5-94d4-15a01315912c","Type":"ContainerDied","Data":"8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72"}
Oct 31 20:02:42.372171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.372104    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk" event={"ID":"9d701ad4-cb46-46e5-94d4-15a01315912c","Type":"ContainerDied","Data":"cf8fc1f28d526c5b63cb96a34a3d18ff149a3828f088da1a4679a2e9aa9ac50f"}
Oct 31 20:02:42.372171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.372130    5958 scope.go:122] "RemoveContainer" containerID="8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72"
Oct 31 20:02:42.385750 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.385731    5958 scope.go:122] "RemoveContainer" containerID="8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72"
Oct 31 20:02:42.386081 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:42.386059    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72\": container with ID starting with 8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72 not found: ID does not exist" containerID="8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72"
Oct 31 20:02:42.386137 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.386090    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72"} err="failed to get container status \"8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72\": rpc error: code = NotFound desc = could not find container \"8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72\": container with ID starting with 8e5715a4b4bb14565c8da38a9565e9fb7ca96c91f03ce8bf4eea27d41d8a7b72 not found: ID does not exist"
Oct 31 20:02:42.406919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.406873    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"]
Oct 31 20:02:42.411245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.411212    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-6hdmk"]
Oct 31 20:02:42.668678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:42.668645    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" path="/var/lib/kubelet/pods/9d701ad4-cb46-46e5-94d4-15a01315912c/volumes"
Oct 31 20:02:43.337179 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:43.337151    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:02:47.050824 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:47.050776    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:02:48.103721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:48.103686    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"]
Oct 31 20:02:48.121185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:48.121150    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"]
Oct 31 20:02:53.113571 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.106862    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw" podUID="c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" containerName="payload-processing" containerID="cri-o://04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b" gracePeriod=25
Oct 31 20:02:53.128467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.127415    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc" podUID="a795c033-7053-445a-b2c9-b6664c756aae" containerName="payload-pre-processing" containerID="cri-o://a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e" gracePeriod=25
Oct 31 20:02:53.499307 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.499278    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:02:53.504659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.504621    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:02:53.518867 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.518828    5958 generic.go:356] "Generic (PLEG): container finished" podID="a795c033-7053-445a-b2c9-b6664c756aae" containerID="a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e" exitCode=0
Oct 31 20:02:53.519183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.518911    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"
Oct 31 20:02:53.519183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.518921    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc" event={"ID":"a795c033-7053-445a-b2c9-b6664c756aae","Type":"ContainerDied","Data":"a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e"}
Oct 31 20:02:53.519183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.518989    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc" event={"ID":"a795c033-7053-445a-b2c9-b6664c756aae","Type":"ContainerDied","Data":"70b8215da3985bf489494bb8343d45e113e2a208fe69b99f4da9e98ebbbe7820"}
Oct 31 20:02:53.519183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.519022    5958 scope.go:122] "RemoveContainer" containerID="a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e"
Oct 31 20:02:53.538621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.538585    5958 generic.go:356] "Generic (PLEG): container finished" podID="c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" containerID="04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b" exitCode=0
Oct 31 20:02:53.539059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.538786    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw" event={"ID":"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0","Type":"ContainerDied","Data":"04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b"}
Oct 31 20:02:53.539059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.538846    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw" event={"ID":"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0","Type":"ContainerDied","Data":"405ed6c2dadf86e801870bf4c2e2a53197d7dcd149fc5bdc2249b9e8f845b976"}
Oct 31 20:02:53.539059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.538933    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"
Oct 31 20:02:53.579530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.579507    5958 scope.go:122] "RemoveContainer" containerID="a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e"
Oct 31 20:02:53.580015 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:53.579983    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e\": container with ID starting with a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e not found: ID does not exist" containerID="a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e"
Oct 31 20:02:53.580195 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.580022    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e"} err="failed to get container status \"a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e\": rpc error: code = NotFound desc = could not find container \"a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e\": container with ID starting with a7ff3a1db2d3965efcd9388a5e268c58e2162a348b06273461927f3b141c725e not found: ID does not exist"
Oct 31 20:02:53.580195 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.580049    5958 scope.go:122] "RemoveContainer" containerID="04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b"
Oct 31 20:02:53.605105 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.605079    5958 scope.go:122] "RemoveContainer" containerID="04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b"
Oct 31 20:02:53.605759 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:02:53.605736    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b\": container with ID starting with 04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b not found: ID does not exist" containerID="04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b"
Oct 31 20:02:53.608086 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.605892    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b"} err="failed to get container status \"04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b\": rpc error: code = NotFound desc = could not find container \"04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b\": container with ID starting with 04063ba949fd346886e799db4ee5e2b64c4a7156e58e868864f9a1dafe76864b not found: ID does not exist"
Oct 31 20:02:53.650434 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.650391    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume\") pod \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") "
Oct 31 20:02:53.650659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.650539    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn\" (UniqueName: \"kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn\") pod \"a795c033-7053-445a-b2c9-b6664c756aae\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") "
Oct 31 20:02:53.650809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.650793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x\" (UniqueName: \"kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x\") pod \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\" (UID: \"c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0\") "
Oct 31 20:02:53.650862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.650851    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume\") pod \"a795c033-7053-445a-b2c9-b6664c756aae\" (UID: \"a795c033-7053-445a-b2c9-b6664c756aae\") "
Oct 31 20:02:53.651009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.650847    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume" pod "c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" (UID: "c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:53.651344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.651313    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume" pod "a795c033-7053-445a-b2c9-b6664c756aae" (UID: "a795c033-7053-445a-b2c9-b6664c756aae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:53.651586 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.651567    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:53.651685 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.651591    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a795c033-7053-445a-b2c9-b6664c756aae-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:53.653201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.653176    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn" pod "a795c033-7053-445a-b2c9-b6664c756aae" (UID: "a795c033-7053-445a-b2c9-b6664c756aae"). InnerVolumeSpecName "kube-api-access-fvbxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:53.653463 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.653442    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x" pod "c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" (UID: "c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0"). InnerVolumeSpecName "kube-api-access-dgj4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:53.752947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.752859    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvbxn\" (UniqueName: \"kubernetes.io/projected/a795c033-7053-445a-b2c9-b6664c756aae-kube-api-access-fvbxn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:53.752947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.752890    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgj4x\" (UniqueName: \"kubernetes.io/projected/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0-kube-api-access-dgj4x\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:53.855193 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.855156    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"]
Oct 31 20:02:53.862442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.862402    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-5cccb98579-hs4dc"]
Oct 31 20:02:53.883818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.883769    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"]
Oct 31 20:02:53.887119 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:53.887088    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-7457796869-bbpbw"]
Oct 31 20:02:54.687701 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:54.687664    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a795c033-7053-445a-b2c9-b6664c756aae" path="/var/lib/kubelet/pods/a795c033-7053-445a-b2c9-b6664c756aae/volumes"
Oct 31 20:02:54.688214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:02:54.688199    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" path="/var/lib/kubelet/pods/c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0/volumes"
Oct 31 20:03:00.393278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:00.393243    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"]
Oct 31 20:03:00.411596 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:00.411555    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"]
Oct 31 20:03:04.940118 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:04.940068    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"]
Oct 31 20:03:04.940648 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:04.940343    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy" containerID="cri-o://3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111" gracePeriod=30
Oct 31 20:03:05.394600 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.394503    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p" podUID="1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" containerName="payload-processing" containerID="cri-o://51612c14c7b4fee2f67de46700d3f8d2218be532fb95190ab6eff357991f004b" gracePeriod=25
Oct 31 20:03:05.412396 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.412343    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f" podUID="9b94853c-f3f6-4f54-994b-218aff398894" containerName="payload-pre-processing" containerID="cri-o://afeb9e1a65e3725b7fd26048f965795391117399fc9b1c1dfd8cccc8a9af3601" gracePeriod=25
Oct 31 20:03:05.722601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.719943    5958 generic.go:356] "Generic (PLEG): container finished" podID="9b94853c-f3f6-4f54-994b-218aff398894" containerID="afeb9e1a65e3725b7fd26048f965795391117399fc9b1c1dfd8cccc8a9af3601" exitCode=0
Oct 31 20:03:05.722601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.720198    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f" event={"ID":"9b94853c-f3f6-4f54-994b-218aff398894","Type":"ContainerDied","Data":"afeb9e1a65e3725b7fd26048f965795391117399fc9b1c1dfd8cccc8a9af3601"}
Oct 31 20:03:05.725289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.725251    5958 generic.go:356] "Generic (PLEG): container finished" podID="1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" containerID="51612c14c7b4fee2f67de46700d3f8d2218be532fb95190ab6eff357991f004b" exitCode=0
Oct 31 20:03:05.725492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.725473    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p" event={"ID":"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f","Type":"ContainerDied","Data":"51612c14c7b4fee2f67de46700d3f8d2218be532fb95190ab6eff357991f004b"}
Oct 31 20:03:05.821718 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.821678    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:03:05.851137 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.851109    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:03:05.976681 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.976594    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume\") pod \"9b94853c-f3f6-4f54-994b-218aff398894\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") "
Oct 31 20:03:05.977087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.976722    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7\" (UniqueName: \"kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7\") pod \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") "
Oct 31 20:03:05.977087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.976929    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k\" (UniqueName: \"kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k\") pod \"9b94853c-f3f6-4f54-994b-218aff398894\" (UID: \"9b94853c-f3f6-4f54-994b-218aff398894\") "
Oct 31 20:03:05.977087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.976997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume\") pod \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\" (UID: \"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f\") "
Oct 31 20:03:05.977087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.977038    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume" pod "9b94853c-f3f6-4f54-994b-218aff398894" (UID: "9b94853c-f3f6-4f54-994b-218aff398894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:05.977451 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.977419    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume" pod "1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" (UID: "1bb54d47-d2ee-4b55-9f85-a8c83752cf1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:05.977795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.977769    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:05.977893 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.977800    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b94853c-f3f6-4f54-994b-218aff398894-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:05.978977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.978953    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7" pod "1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" (UID: "1bb54d47-d2ee-4b55-9f85-a8c83752cf1f"). InnerVolumeSpecName "kube-api-access-58zt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:05.979218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:05.979193    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k" pod "9b94853c-f3f6-4f54-994b-218aff398894" (UID: "9b94853c-f3f6-4f54-994b-218aff398894"). InnerVolumeSpecName "kube-api-access-mzr5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:06.079952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.079916    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zt7\" (UniqueName: \"kubernetes.io/projected/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f-kube-api-access-58zt7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:06.079952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.079949    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzr5k\" (UniqueName: \"kubernetes.io/projected/9b94853c-f3f6-4f54-994b-218aff398894-kube-api-access-mzr5k\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:06.735870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.735839    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"
Oct 31 20:03:06.736137 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.735871    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f" event={"ID":"9b94853c-f3f6-4f54-994b-218aff398894","Type":"ContainerDied","Data":"d561c5fe886bf87725bd6586bf050132d11fb5734e11f4a391adf9703516cd8f"}
Oct 31 20:03:06.736137 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.735955    5958 scope.go:122] "RemoveContainer" containerID="afeb9e1a65e3725b7fd26048f965795391117399fc9b1c1dfd8cccc8a9af3601"
Oct 31 20:03:06.738196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.738176    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"
Oct 31 20:03:06.738301 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.738210    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p" event={"ID":"1bb54d47-d2ee-4b55-9f85-a8c83752cf1f","Type":"ContainerDied","Data":"ddb01521173ea877eea55cbee72ec8431aed3c898d08b9e8bd7f8bede91377e2"}
Oct 31 20:03:06.748923 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.748904    5958 scope.go:122] "RemoveContainer" containerID="51612c14c7b4fee2f67de46700d3f8d2218be532fb95190ab6eff357991f004b"
Oct 31 20:03:06.766885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.766856    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"]
Oct 31 20:03:06.774762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.774734    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-bdbd55747-zkl6p"]
Oct 31 20:03:06.791484 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.791454    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"]
Oct 31 20:03:06.800501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:06.800472    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-6f899ff859-5685f"]
Oct 31 20:03:08.667344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:08.667309    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" path="/var/lib/kubelet/pods/1bb54d47-d2ee-4b55-9f85-a8c83752cf1f/volumes"
Oct 31 20:03:08.667907 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:08.667876    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b94853c-f3f6-4f54-994b-218aff398894" path="/var/lib/kubelet/pods/9b94853c-f3f6-4f54-994b-218aff398894/volumes"
Oct 31 20:03:10.278436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.278410    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 20:03:10.366719 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.366667    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.366933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.366730    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.366933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.366772    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.366933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.366846    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.367332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.367173    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.367332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.367268    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.367524 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.367489    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.367779 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.367750    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.367972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.367916    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:10.368157 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.368141    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5844ffbc-e285-4de7-9af7-b6a384bc398b-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.368157 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.368159    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.368291 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.368170    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.369604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.369522    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.369604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.369565    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:10.369812 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.369674    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "kube-api-access-j56pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:10.394813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.394776    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"]
Oct 31 20:03:10.410800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.410763    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"]
Oct 31 20:03:10.469054 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.469017    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.469355 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.469336    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.469463 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.469371    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket\") pod \"5844ffbc-e285-4de7-9af7-b6a384bc398b\" (UID: \"5844ffbc-e285-4de7-9af7-b6a384bc398b\") "
Oct 31 20:03:10.469634 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.469598    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.469777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.469758    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.470104 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.470089    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.470155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.470105    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.470155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.470117    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j56pj\" (UniqueName: \"kubernetes.io/projected/5844ffbc-e285-4de7-9af7-b6a384bc398b-kube-api-access-j56pj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.470155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.470127    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.470155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.470136    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5844ffbc-e285-4de7-9af7-b6a384bc398b-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.471191 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.471170    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo" pod "5844ffbc-e285-4de7-9af7-b6a384bc398b" (UID: "5844ffbc-e285-4de7-9af7-b6a384bc398b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:03:10.572499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.572392    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5844ffbc-e285-4de7-9af7-b6a384bc398b-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.777420 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.777384    5958 generic.go:356] "Generic (PLEG): container finished" podID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerID="3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111" exitCode=0
Oct 31 20:03:10.777650 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.777441    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"
Oct 31 20:03:10.777650 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.777466    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" event={"ID":"5844ffbc-e285-4de7-9af7-b6a384bc398b","Type":"ContainerDied","Data":"3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111"}
Oct 31 20:03:10.777650 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.777522    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr" event={"ID":"5844ffbc-e285-4de7-9af7-b6a384bc398b","Type":"ContainerDied","Data":"e05c5057cdb2b33cefb0493e2720207b06dc8785d07da8cba36fbf068b8f73af"}
Oct 31 20:03:10.777650 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.777581    5958 scope.go:122] "RemoveContainer" containerID="3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111"
Oct 31 20:03:10.789339 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.789322    5958 scope.go:122] "RemoveContainer" containerID="3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111"
Oct 31 20:03:10.789649 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:10.789629    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111\": container with ID starting with 3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111 not found: ID does not exist" containerID="3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111"
Oct 31 20:03:10.789704 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.789657    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111"} err="failed to get container status \"3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111\": rpc error: code = NotFound desc = could not find container \"3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111\": container with ID starting with 3b01db63157c57d411c04ab2ab507da204c118eec0922cfedb719626b823f111 not found: ID does not exist"
Oct 31 20:03:10.802177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.802143    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"]
Oct 31 20:03:10.806368 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:10.806331    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-qqcjr"]
Oct 31 20:03:12.677784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:12.677750    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" path="/var/lib/kubelet/pods/5844ffbc-e285-4de7-9af7-b6a384bc398b/volumes"
Oct 31 20:03:15.163845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:15.163801    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:03:15.872759 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:15.872694    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"]
Oct 31 20:03:15.894527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:15.894490    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"]
Oct 31 20:03:17.049317 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:17.049276    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:03:19.383494 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.383460    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"]
Oct 31 20:03:19.384599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384573    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy"
Oct 31 20:03:19.384599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384599    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384613    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384620    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384640    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" containerName="payload-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384648    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" containerName="payload-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384668    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9b94853c-f3f6-4f54-994b-218aff398894" containerName="payload-pre-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384676    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b94853c-f3f6-4f54-994b-218aff398894" containerName="payload-pre-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384696    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a795c033-7053-445a-b2c9-b6664c756aae" containerName="payload-pre-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384704    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a795c033-7053-445a-b2c9-b6664c756aae" containerName="payload-pre-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384750    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" containerName="payload-processing"
Oct 31 20:03:19.384799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.384757    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" containerName="payload-processing"
Oct 31 20:03:19.385315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.385008    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb54d47-d2ee-4b55-9f85-a8c83752cf1f" containerName="payload-processing"
Oct 31 20:03:19.385315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.385027    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b94853c-f3f6-4f54-994b-218aff398894" containerName="payload-pre-processing"
Oct 31 20:03:19.385315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.385039    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a795c033-7053-445a-b2c9-b6664c756aae" containerName="payload-pre-processing"
Oct 31 20:03:19.385315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.385060    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c5d5b3-ab78-4dbe-ae7d-b1bfe489e5e0" containerName="payload-processing"
Oct 31 20:03:19.385315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.385082    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d701ad4-cb46-46e5-94d4-15a01315912c" containerName="istio-proxy"
Oct 31 20:03:19.385315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.385092    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5844ffbc-e285-4de7-9af7-b6a384bc398b" containerName="istio-proxy"
Oct 31 20:03:19.390898 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.390875    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.406013 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.405986    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a84fcb-dockercfg-gsxtm\""
Oct 31 20:03:19.406945 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.406922    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"]
Oct 31 20:03:19.462974 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.453671    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a84fcb\""
Oct 31 20:03:19.507569 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.503223    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbwk\" (UniqueName: \"kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.507569 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.503505    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.609877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.609833    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.610200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.610166    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbwk\" (UniqueName: \"kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.611745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.611717    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.625639 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.625587    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbwk\" (UniqueName: \"kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk\") pod \"payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:19.713590 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:19.712189    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:20.015884 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:03:20.015842    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37467c1d_a0f5_462d_a9ee_4e98ef27713f.slice/crio-13cc37b853529519b835940f4f67e90f1c9cb438f75c8848dab3e222fc04905d WatchSource:0}: Error finding container 13cc37b853529519b835940f4f67e90f1c9cb438f75c8848dab3e222fc04905d: Status 404 returned error can't find the container with id 13cc37b853529519b835940f4f67e90f1c9cb438f75c8848dab3e222fc04905d
Oct 31 20:03:20.016486 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.016455    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"]
Oct 31 20:03:20.874927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.874889    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" containerName="payload-processing" containerID="cri-o://9598d85ed69fb7ca3f5ac276354f45e8ed46f7c638d4d073065cb9efa247f9be" gracePeriod=25
Oct 31 20:03:20.895190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.895108    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" containerName="payload-pre-processing" containerID="cri-o://5aa3c1e94be400b25464f3e90355dd67635d503df994806d5b1493cc74e405c5" gracePeriod=25
Oct 31 20:03:20.936629 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.936589    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf" event={"ID":"37467c1d-a0f5-462d-a9ee-4e98ef27713f","Type":"ContainerStarted","Data":"3de88dd0f80b4c15cf671beea152366bbf2e66d6b1d37ad84ca2c697adc1dac9"}
Oct 31 20:03:20.936823 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.936661    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:20.936823 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.936690    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf" event={"ID":"37467c1d-a0f5-462d-a9ee-4e98ef27713f","Type":"ContainerStarted","Data":"13cc37b853529519b835940f4f67e90f1c9cb438f75c8848dab3e222fc04905d"}
Oct 31 20:03:20.961314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:20.961261    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf" podStartSLOduration=1.961246391 podStartE2EDuration="1.961246391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:20.958062038 +0000 UTC m=+2594.419436657" watchObservedRunningTime="2026-10-31 20:03:20.961246391 +0000 UTC m=+2594.422621005"
Oct 31 20:03:22.427336 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:22.427290    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:03:22.427878 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:22.427494    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:03:25.420125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.420083    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" containerID="cri-o://bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e" gracePeriod=45
Oct 31 20:03:25.439077 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.439039    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" containerID="cri-o://65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9" gracePeriod=45
Oct 31 20:03:25.619696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.617971    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused"
Oct 31 20:03:25.925536 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.925504    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 20:03:25.950599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.950508    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 20:03:25.995597 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.995554    5958 generic.go:356] "Generic (PLEG): container finished" podID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerID="65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9" exitCode=0
Oct 31 20:03:25.995597 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.995576    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" event={"ID":"f54649bb-ad77-4af9-ae4c-16e9aa03f68e","Type":"ContainerDied","Data":"65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9"}
Oct 31 20:03:25.996246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.995629    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82" event={"ID":"f54649bb-ad77-4af9-ae4c-16e9aa03f68e","Type":"ContainerDied","Data":"20958a41a598e50ae5fe7919d3ba0ff5c5c84eb1a4c51fed9af5612e23e39b2c"}
Oct 31 20:03:25.996246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.995655    5958 scope.go:122] "RemoveContainer" containerID="65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9"
Oct 31 20:03:25.996430 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.996407    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"
Oct 31 20:03:25.998272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.998248    5958 generic.go:356] "Generic (PLEG): container finished" podID="5f56a551-769a-4653-bef3-dd83b782d8da" containerID="bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e" exitCode=0
Oct 31 20:03:25.998386 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.998303    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"
Oct 31 20:03:25.998462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.998305    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" event={"ID":"5f56a551-769a-4653-bef3-dd83b782d8da","Type":"ContainerDied","Data":"bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e"}
Oct 31 20:03:25.998517 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:25.998485    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" event={"ID":"5f56a551-769a-4653-bef3-dd83b782d8da","Type":"ContainerDied","Data":"a6d6903c0a73b7238ba6260f52523d171aec8fc308b84e9e762c45efefe6889b"}
Oct 31 20:03:26.010727 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.010703    5958 scope.go:122] "RemoveContainer" containerID="65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9"
Oct 31 20:03:26.011078 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:26.011053    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9\": container with ID starting with 65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9 not found: ID does not exist" containerID="65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9"
Oct 31 20:03:26.011193 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.011174    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9"} err="failed to get container status \"65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9\": rpc error: code = NotFound desc = could not find container \"65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9\": container with ID starting with 65e92b078737d488040a66183180aa6da9a0a54f27d7545fd507985483d995c9 not found: ID does not exist"
Oct 31 20:03:26.011262 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.011254    5958 scope.go:122] "RemoveContainer" containerID="bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e"
Oct 31 20:03:26.024490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.024463    5958 scope.go:122] "RemoveContainer" containerID="bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e"
Oct 31 20:03:26.024891 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:26.024869    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e\": container with ID starting with bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e not found: ID does not exist" containerID="bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e"
Oct 31 20:03:26.024974 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.024900    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e"} err="failed to get container status \"bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e\": rpc error: code = NotFound desc = could not find container \"bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e\": container with ID starting with bb15bd34ca964c086d125e5e062a400c124e121c6df4f0a61db5f66bf80af04e not found: ID does not exist"
Oct 31 20:03:26.031831 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.031805    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home\") pod \"5f56a551-769a-4653-bef3-dd83b782d8da\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") "
Oct 31 20:03:26.032045 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.031897    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm\") pod \"5f56a551-769a-4653-bef3-dd83b782d8da\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") "
Oct 31 20:03:26.032045 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.031982    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9\" (UniqueName: \"kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9\") pod \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") "
Oct 31 20:03:26.032045 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032035    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir\") pod \"5f56a551-769a-4653-bef3-dd83b782d8da\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") "
Oct 31 20:03:26.032244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032070    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache\") pod \"5f56a551-769a-4653-bef3-dd83b782d8da\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") "
Oct 31 20:03:26.032244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032141    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm\") pod \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") "
Oct 31 20:03:26.032244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032187    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home\") pod \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") "
Oct 31 20:03:26.032412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032326    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs\") pod \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") "
Oct 31 20:03:26.032471 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032412    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home" pod "5f56a551-769a-4653-bef3-dd83b782d8da" (UID: "5f56a551-769a-4653-bef3-dd83b782d8da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.032471 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032440    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache\") pod \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") "
Oct 31 20:03:26.032599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032501    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x\" (UniqueName: \"kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x\") pod \"5f56a551-769a-4653-bef3-dd83b782d8da\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") "
Oct 31 20:03:26.032599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032577    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir\") pod \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\" (UID: \"f54649bb-ad77-4af9-ae4c-16e9aa03f68e\") "
Oct 31 20:03:26.032717 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.032608    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs\") pod \"5f56a551-769a-4653-bef3-dd83b782d8da\" (UID: \"5f56a551-769a-4653-bef3-dd83b782d8da\") "
Oct 31 20:03:26.033752 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.033088    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir" pod "5f56a551-769a-4653-bef3-dd83b782d8da" (UID: "5f56a551-769a-4653-bef3-dd83b782d8da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.033752 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.033296    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache" pod "f54649bb-ad77-4af9-ae4c-16e9aa03f68e" (UID: "f54649bb-ad77-4af9-ae4c-16e9aa03f68e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.033752 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.033477    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home" pod "f54649bb-ad77-4af9-ae4c-16e9aa03f68e" (UID: "f54649bb-ad77-4af9-ae4c-16e9aa03f68e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.034140 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.034115    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.034216 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.034157    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.034216 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.034171    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.034216 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.034184    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.035420 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.035196    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache" pod "5f56a551-769a-4653-bef3-dd83b782d8da" (UID: "5f56a551-769a-4653-bef3-dd83b782d8da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.035420 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.035361    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir" pod "f54649bb-ad77-4af9-ae4c-16e9aa03f68e" (UID: "f54649bb-ad77-4af9-ae4c-16e9aa03f68e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.035634 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.035565    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9" pod "f54649bb-ad77-4af9-ae4c-16e9aa03f68e" (UID: "f54649bb-ad77-4af9-ae4c-16e9aa03f68e"). InnerVolumeSpecName "kube-api-access-djlp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:26.037891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.037200    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs" pod "5f56a551-769a-4653-bef3-dd83b782d8da" (UID: "5f56a551-769a-4653-bef3-dd83b782d8da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:26.037891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.037338    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm" pod "f54649bb-ad77-4af9-ae4c-16e9aa03f68e" (UID: "f54649bb-ad77-4af9-ae4c-16e9aa03f68e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.038334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.038272    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs" pod "f54649bb-ad77-4af9-ae4c-16e9aa03f68e" (UID: "f54649bb-ad77-4af9-ae4c-16e9aa03f68e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:26.039114 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.039090    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm" pod "5f56a551-769a-4653-bef3-dd83b782d8da" (UID: "5f56a551-769a-4653-bef3-dd83b782d8da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:26.039249 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.039227    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x" pod "5f56a551-769a-4653-bef3-dd83b782d8da" (UID: "5f56a551-769a-4653-bef3-dd83b782d8da"). InnerVolumeSpecName "kube-api-access-pdj4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148598    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148640    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdj4x\" (UniqueName: \"kubernetes.io/projected/5f56a551-769a-4653-bef3-dd83b782d8da-kube-api-access-pdj4x\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148656    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148671    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f56a551-769a-4653-bef3-dd83b782d8da-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148684    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148698    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djlp9\" (UniqueName: \"kubernetes.io/projected/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-kube-api-access-djlp9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148715    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f56a551-769a-4653-bef3-dd83b782d8da-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.150686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.148728    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54649bb-ad77-4af9-ae4c-16e9aa03f68e-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:26.334132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.334096    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"]
Oct 31 20:03:26.340697 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.340664    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/premium-w0-23843f-kserve-5c79958988-g7h82"]
Oct 31 20:03:26.353689 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.353657    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"]
Oct 31 20:03:26.357986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.357954    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz"]
Oct 31 20:03:26.667111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.667074    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" path="/var/lib/kubelet/pods/5f56a551-769a-4653-bef3-dd83b782d8da/volumes"
Oct 31 20:03:26.667616 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.667597    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" path="/var/lib/kubelet/pods/f54649bb-ad77-4af9-ae4c-16e9aa03f68e/volumes"
Oct 31 20:03:26.850196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:26.850085    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-2-w0-23843f-kserve-b6c9c6768-48lgz" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:03:31.947614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:31.947523    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:03:32.426979 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.426934    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:03:32.427177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.427064    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:03:32.777774 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.777694    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"]
Oct 31 20:03:32.778803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.778768    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main"
Oct 31 20:03:32.778803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.778802    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main"
Oct 31 20:03:32.779026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.778886    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main"
Oct 31 20:03:32.779026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.778895    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main"
Oct 31 20:03:32.779118 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.779106    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f56a551-769a-4653-bef3-dd83b782d8da" containerName="main"
Oct 31 20:03:32.779155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.779127    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54649bb-ad77-4af9-ae4c-16e9aa03f68e" containerName="main"
Oct 31 20:03:32.782785 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.782768    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.792624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.792596    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-435728-openshift-default-dockercfg-sdb9g\""
Oct 31 20:03:32.800361 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.800336    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"]
Oct 31 20:03:32.887525 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887489    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.887525 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887527    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.887813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887569    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.887813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887668    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.887920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887800    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.887920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887861    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.888004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887925    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.888004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.887969    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.888101 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.888004    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwg9\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.989417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989379    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.989909 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989434    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.989909 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989469    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.989909 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989501    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.989909 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989630    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwg9\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989953    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989977    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.989995    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.990015    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.990051    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.990085    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.990381    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.990418    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.990464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.990457    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.991866 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.991848    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:32.992438 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:32.992416    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:33.003209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.003181    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:33.003420 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.003395    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwg9\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:33.101799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.101715    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:33.397991 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:03:33.397959    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b11505_5149_4b74_b311_e6399e2925ec.slice/crio-4ea51d4e45e1ad3db780c6611ef265f87ca055481a5a3140b3a539dd12cf99da WatchSource:0}: Error finding container 4ea51d4e45e1ad3db780c6611ef265f87ca055481a5a3140b3a539dd12cf99da: Status 404 returned error can't find the container with id 4ea51d4e45e1ad3db780c6611ef265f87ca055481a5a3140b3a539dd12cf99da
Oct 31 20:03:33.399235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.399207    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"]
Oct 31 20:03:33.400683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.400648    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:03:33.400786 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.400729    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:03:33.400786 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:33.400780    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:03:34.125507 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:34.125464    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" event={"ID":"b1b11505-5149-4b74-b311-e6399e2925ec","Type":"ContainerStarted","Data":"6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b"}
Oct 31 20:03:34.126003 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:34.125534    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" event={"ID":"b1b11505-5149-4b74-b311-e6399e2925ec","Type":"ContainerStarted","Data":"4ea51d4e45e1ad3db780c6611ef265f87ca055481a5a3140b3a539dd12cf99da"}
Oct 31 20:03:34.154899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:34.154842    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podStartSLOduration=2.154825871 podStartE2EDuration="2.154825871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:34.153341656 +0000 UTC m=+2607.614716268" watchObservedRunningTime="2026-10-31 20:03:34.154825871 +0000 UTC m=+2607.616200485"
Oct 31 20:03:35.102202 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:35.102167    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:35.113742 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:35.113702    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 20:03:35.114041 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:35.114011    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 20:03:36.102615 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:36.102561    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 20:03:36.103199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:36.102645    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 20:03:37.103038 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:37.102998    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 20:03:37.103467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:37.103064    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 20:03:38.103181 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:38.103142    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 20:03:38.103675 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:38.103203    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 20:03:39.102132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:39.102095    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 20:03:39.102326 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:39.102162    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 20:03:40.102929 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:40.102890    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body=
Oct 31 20:03:40.103469 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:40.102963    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused"
Oct 31 20:03:41.108306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.108273    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:41.109654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.109584    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:41.110312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.110280    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:03:41.201351 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.201025    5958 generic.go:356] "Generic (PLEG): container finished" podID="07738ae8-6f04-404f-80ef-06031d966bfb" containerID="9598d85ed69fb7ca3f5ac276354f45e8ed46f7c638d4d073065cb9efa247f9be" exitCode=0
Oct 31 20:03:41.201351 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.201304    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" event={"ID":"07738ae8-6f04-404f-80ef-06031d966bfb","Type":"ContainerDied","Data":"9598d85ed69fb7ca3f5ac276354f45e8ed46f7c638d4d073065cb9efa247f9be"}
Oct 31 20:03:41.201639 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.201361    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq" event={"ID":"07738ae8-6f04-404f-80ef-06031d966bfb","Type":"ContainerDied","Data":"11db9ff8c43c94ba387007eb85d9f67d4d3cc5ba07d680fcd6a8be739982e61f"}
Oct 31 20:03:41.201639 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.201387    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11db9ff8c43c94ba387007eb85d9f67d4d3cc5ba07d680fcd6a8be739982e61f"
Oct 31 20:03:41.214495 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.214459    5958 generic.go:356] "Generic (PLEG): container finished" podID="a65915d4-8e42-45ab-abc6-b539629821cf" containerID="5aa3c1e94be400b25464f3e90355dd67635d503df994806d5b1493cc74e405c5" exitCode=0
Oct 31 20:03:41.217863 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.217800    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" event={"ID":"a65915d4-8e42-45ab-abc6-b539629821cf","Type":"ContainerDied","Data":"5aa3c1e94be400b25464f3e90355dd67635d503df994806d5b1493cc74e405c5"}
Oct 31 20:03:41.224609 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.224533    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 20:03:41.292450 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.292398    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 20:03:41.398410 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.398369    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882\" (UniqueName: \"kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882\") pod \"a65915d4-8e42-45ab-abc6-b539629821cf\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") "
Oct 31 20:03:41.398657 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.398425    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume\") pod \"07738ae8-6f04-404f-80ef-06031d966bfb\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") "
Oct 31 20:03:41.398657 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.398581    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume\") pod \"a65915d4-8e42-45ab-abc6-b539629821cf\" (UID: \"a65915d4-8e42-45ab-abc6-b539629821cf\") "
Oct 31 20:03:41.398657 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.398627    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l\" (UniqueName: \"kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l\") pod \"07738ae8-6f04-404f-80ef-06031d966bfb\" (UID: \"07738ae8-6f04-404f-80ef-06031d966bfb\") "
Oct 31 20:03:41.398937 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.398908    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume" pod "07738ae8-6f04-404f-80ef-06031d966bfb" (UID: "07738ae8-6f04-404f-80ef-06031d966bfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:41.399021 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.398965    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume" pod "a65915d4-8e42-45ab-abc6-b539629821cf" (UID: "a65915d4-8e42-45ab-abc6-b539629821cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:41.399646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.399624    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65915d4-8e42-45ab-abc6-b539629821cf-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:41.399646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.399646    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07738ae8-6f04-404f-80ef-06031d966bfb-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:41.400782 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.400758    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l" pod "07738ae8-6f04-404f-80ef-06031d966bfb" (UID: "07738ae8-6f04-404f-80ef-06031d966bfb"). InnerVolumeSpecName "kube-api-access-ljm2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:41.400882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.400866    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882" pod "a65915d4-8e42-45ab-abc6-b539629821cf" (UID: "a65915d4-8e42-45ab-abc6-b539629821cf"). InnerVolumeSpecName "kube-api-access-d4882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:41.501029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.500925    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4882\" (UniqueName: \"kubernetes.io/projected/a65915d4-8e42-45ab-abc6-b539629821cf-kube-api-access-d4882\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:41.501029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:41.500962    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljm2l\" (UniqueName: \"kubernetes.io/projected/07738ae8-6f04-404f-80ef-06031d966bfb-kube-api-access-ljm2l\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:42.232081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.231992    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"
Oct 31 20:03:42.232081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.232042    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz" event={"ID":"a65915d4-8e42-45ab-abc6-b539629821cf","Type":"ContainerDied","Data":"ee518bcc84edfee9c44a6148ac9cc12a8e6b7621b2c200821d3a886d1aff4a3c"}
Oct 31 20:03:42.233236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.232108    5958 scope.go:122] "RemoveContainer" containerID="5aa3c1e94be400b25464f3e90355dd67635d503df994806d5b1493cc74e405c5"
Oct 31 20:03:42.233236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.232320    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"
Oct 31 20:03:42.272701 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.272661    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"]
Oct 31 20:03:42.277195 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.277171    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-23843f-6dcc5f75c5-jr2rz"]
Oct 31 20:03:42.294558 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.294509    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"]
Oct 31 20:03:42.300410 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.300376    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-23843f-66cb64b99f-v9xjq"]
Oct 31 20:03:42.667200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.667166    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" path="/var/lib/kubelet/pods/07738ae8-6f04-404f-80ef-06031d966bfb/volumes"
Oct 31 20:03:42.667678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:42.667664    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" path="/var/lib/kubelet/pods/a65915d4-8e42-45ab-abc6-b539629821cf/volumes"
Oct 31 20:03:44.073404 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.073367    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"]
Oct 31 20:03:44.076047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.073656    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r" podUID="2c686a0b-589f-4436-a395-ad20a393e79c" containerName="maas-api" containerID="cri-o://52dfac1cecb70f5082015d89209ee24b56a6450be8710a7133a18377b90ff471" gracePeriod=30
Oct 31 20:03:44.208351 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:44.208261    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c686a0b_589f_4436_a395_ad20a393e79c.slice/crio-conmon-52dfac1cecb70f5082015d89209ee24b56a6450be8710a7133a18377b90ff471.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:03:44.259169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.259124    5958 generic.go:356] "Generic (PLEG): container finished" podID="2c686a0b-589f-4436-a395-ad20a393e79c" containerID="52dfac1cecb70f5082015d89209ee24b56a6450be8710a7133a18377b90ff471" exitCode=0
Oct 31 20:03:44.259375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.259195    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r" event={"ID":"2c686a0b-589f-4436-a395-ad20a393e79c","Type":"ContainerDied","Data":"52dfac1cecb70f5082015d89209ee24b56a6450be8710a7133a18377b90ff471"}
Oct 31 20:03:44.554579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.551047    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:03:44.586572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.585422    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca\") pod \"2c686a0b-589f-4436-a395-ad20a393e79c\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") "
Oct 31 20:03:44.586572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.585636    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls\") pod \"2c686a0b-589f-4436-a395-ad20a393e79c\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") "
Oct 31 20:03:44.586572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.585682    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws\" (UniqueName: \"kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws\") pod \"2c686a0b-589f-4436-a395-ad20a393e79c\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") "
Oct 31 20:03:44.586572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.585730    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs\") pod \"2c686a0b-589f-4436-a395-ad20a393e79c\" (UID: \"2c686a0b-589f-4436-a395-ad20a393e79c\") "
Oct 31 20:03:44.592733 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.592697    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca" pod "2c686a0b-589f-4436-a395-ad20a393e79c" (UID: "2c686a0b-589f-4436-a395-ad20a393e79c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:44.602671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.598974    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls" pod "2c686a0b-589f-4436-a395-ad20a393e79c" (UID: "2c686a0b-589f-4436-a395-ad20a393e79c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:44.602671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.599121    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs" pod "2c686a0b-589f-4436-a395-ad20a393e79c" (UID: "2c686a0b-589f-4436-a395-ad20a393e79c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:44.602671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.599591    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws" pod "2c686a0b-589f-4436-a395-ad20a393e79c" (UID: "2c686a0b-589f-4436-a395-ad20a393e79c"). InnerVolumeSpecName "kube-api-access-d8mws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:44.689986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.689955    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c686a0b-589f-4436-a395-ad20a393e79c-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:44.690196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.690178    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:44.690311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.690298    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mws\" (UniqueName: \"kubernetes.io/projected/2c686a0b-589f-4436-a395-ad20a393e79c-kube-api-access-d8mws\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:44.690430 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:44.690418    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c686a0b-589f-4436-a395-ad20a393e79c-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.270131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:45.270098    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r" event={"ID":"2c686a0b-589f-4436-a395-ad20a393e79c","Type":"ContainerDied","Data":"5197d07761dd586d9af6997c2e603a971fd85291931e25a4a003157000f741c3"}
Oct 31 20:03:45.270652 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:45.270146    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"
Oct 31 20:03:45.270652 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:45.270164    5958 scope.go:122] "RemoveContainer" containerID="52dfac1cecb70f5082015d89209ee24b56a6450be8710a7133a18377b90ff471"
Oct 31 20:03:45.296463 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:45.296430    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"]
Oct 31 20:03:45.303706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:45.303045    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c734f5-84df75fc9b-d4t5r"]
Oct 31 20:03:46.667797 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:46.667718    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c686a0b-589f-4436-a395-ad20a393e79c" path="/var/lib/kubelet/pods/2c686a0b-589f-4436-a395-ad20a393e79c/volumes"
Oct 31 20:03:47.048022 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:47.047941    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:03:52.904152 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:52.904108    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"]
Oct 31 20:03:52.904612 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:52.904372    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" containerID="cri-o://e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b" gracePeriod=30
Oct 31 20:03:52.912061 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:52.912031    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"]
Oct 31 20:03:52.912267 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:52.912246    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy" containerID="cri-o://3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d" gracePeriod=30
Oct 31 20:03:53.510068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:53.510031    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:03:53.510271 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:53.510095    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:03:58.233104 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.233061    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:03:58.285467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.285432    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:03:58.322577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.322519    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.322816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.322602    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.322816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.322656    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.322816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.322809    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.322999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.322841    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.323081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.323043    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.323168 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.323152    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.323231 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.323210    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.323286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.323269    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token\") pod \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\" (UID: \"e374bf0e-a2e2-49aa-b9a4-b0bb77753193\") "
Oct 31 20:03:58.324981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.324619    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.324981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.324666    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.324981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.324824    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:58.325245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.325014    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.325312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.325251    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.326781 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.326756    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:03:58.327669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.327640    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.327771 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.327695    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.328864 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.328839    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m" pod "e374bf0e-a2e2-49aa-b9a4-b0bb77753193" (UID: "e374bf0e-a2e2-49aa-b9a4-b0bb77753193"). InnerVolumeSpecName "kube-api-access-qkw4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.393773 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.393731    5958 generic.go:356] "Generic (PLEG): container finished" podID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerID="3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d" exitCode=0
Oct 31 20:03:58.393981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.393788    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" event={"ID":"1880b0c3-b0c0-40f4-b21e-3885303d14da","Type":"ContainerDied","Data":"3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d"}
Oct 31 20:03:58.393981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.393840    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"
Oct 31 20:03:58.393981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.393859    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l" event={"ID":"1880b0c3-b0c0-40f4-b21e-3885303d14da","Type":"ContainerDied","Data":"e5b2c958e7c0401a87cf05d36837c62c9400c80d4e5e52d59381b745de2d662f"}
Oct 31 20:03:58.393981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.393891    5958 scope.go:122] "RemoveContainer" containerID="3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d"
Oct 31 20:03:58.396042 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.396014    5958 generic.go:356] "Generic (PLEG): container finished" podID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerID="e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b" exitCode=0
Oct 31 20:03:58.396192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.396071    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" event={"ID":"e374bf0e-a2e2-49aa-b9a4-b0bb77753193","Type":"ContainerDied","Data":"e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b"}
Oct 31 20:03:58.396192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.396089    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"
Oct 31 20:03:58.396192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.396109    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8" event={"ID":"e374bf0e-a2e2-49aa-b9a4-b0bb77753193","Type":"ContainerDied","Data":"cc5972dff86f87f0abc4ef58555827e5454ecb10178c26172cb8a3c7e561f9e1"}
Oct 31 20:03:58.409775 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.409744    5958 scope.go:122] "RemoveContainer" containerID="3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d"
Oct 31 20:03:58.410147 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:58.410126    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d\": container with ID starting with 3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d not found: ID does not exist" containerID="3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d"
Oct 31 20:03:58.410217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.410156    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d"} err="failed to get container status \"3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d\": rpc error: code = NotFound desc = could not find container \"3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d\": container with ID starting with 3dbce038a5421918c70d674ee843a1b511e758ccf4ed293de22e45b13c826b3d not found: ID does not exist"
Oct 31 20:03:58.410217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.410175    5958 scope.go:122] "RemoveContainer" containerID="e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b"
Oct 31 20:03:58.425329 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.425288    5958 scope.go:122] "RemoveContainer" containerID="e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b"
Oct 31 20:03:58.425924 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.425897    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426008 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.425973    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426095 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426081    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426145 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426130    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426184 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426161    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426223 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426201    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426265 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426231    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426300 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426265    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.426386 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.426372    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo\") pod \"1880b0c3-b0c0-40f4-b21e-3885303d14da\" (UID: \"1880b0c3-b0c0-40f4-b21e-3885303d14da\") "
Oct 31 20:03:58.427238 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427213    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427245    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkw4m\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-kube-api-access-qkw4m\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427261    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427273    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427287    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427300    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427493 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427313    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427493 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427327    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.427493 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.427340    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e374bf0e-a2e2-49aa-b9a4-b0bb77753193-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.428426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.428374    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.428672 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.428648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.428833 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.428538    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:58.429051 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.428975    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.429243 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.429217    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.429778 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.429748    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:03:58.429896 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.429880    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.431947 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:03:58.430817    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b\": container with ID starting with e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b not found: ID does not exist" containerID="e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b"
Oct 31 20:03:58.431947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.430861    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b"} err="failed to get container status \"e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b\": rpc error: code = NotFound desc = could not find container \"e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b\": container with ID starting with e473ff92f5686717cdf3da2539b6ca942a61c18baec335c1539ea70f9f265c9b not found: ID does not exist"
Oct 31 20:03:58.431947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.431808    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.432282 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.432262    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj" pod "1880b0c3-b0c0-40f4-b21e-3885303d14da" (UID: "1880b0c3-b0c0-40f4-b21e-3885303d14da"). InnerVolumeSpecName "kube-api-access-cv8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.432447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.432430    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"]
Oct 31 20:03:58.436818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.436796    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-nr5p8"]
Oct 31 20:03:58.529015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.528972    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529011    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529025    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529040    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529054    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529068    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529082    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1880b0c3-b0c0-40f4-b21e-3885303d14da-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529096    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv8sj\" (UniqueName: \"kubernetes.io/projected/1880b0c3-b0c0-40f4-b21e-3885303d14da-kube-api-access-cv8sj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.529245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.529110    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1880b0c3-b0c0-40f4-b21e-3885303d14da-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.666205 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.666168    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" path="/var/lib/kubelet/pods/e374bf0e-a2e2-49aa-b9a4-b0bb77753193/volumes"
Oct 31 20:03:58.718223 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.718141    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"]
Oct 31 20:03:58.739257 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:03:58.739222    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-5552l"]
Oct 31 20:04:00.666588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:00.666537    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" path="/var/lib/kubelet/pods/1880b0c3-b0c0-40f4-b21e-3885303d14da/volumes"
Oct 31 20:04:02.525595 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:02.525558    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:04:09.727100 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:09.727065    5958 scope.go:122] "RemoveContainer" containerID="9598d85ed69fb7ca3f5ac276354f45e8ed46f7c638d4d073065cb9efa247f9be"
Oct 31 20:04:10.841139 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:10.841099    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"]
Oct 31 20:04:10.841695 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:10.841368    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" containerID="cri-o://65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb" gracePeriod=30
Oct 31 20:04:11.734601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:11.734527    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:11.734764 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:11.734639    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:16.225350 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.225312    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 20:04:16.320330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320287    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320409    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320469    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320505    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320571    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.320614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320593    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320644    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320709    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320817    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.320955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320837    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.320955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320840    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.320955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.320900    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy\") pod \"3ed136f9-1352-45bd-9f0b-789ecbd43571\" (UID: \"3ed136f9-1352-45bd-9f0b-789ecbd43571\") "
Oct 31 20:04:16.321334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.321030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.321641 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.321604    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:16.321799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.321655    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.321939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.321911    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.322064 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.321942    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.322064 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.321959    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.323725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.323693    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "kube-api-access-zkbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:16.323836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.323764    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:16.323836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.323800    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:16.324166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.324137    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy" pod "3ed136f9-1352-45bd-9f0b-789ecbd43571" (UID: "3ed136f9-1352-45bd-9f0b-789ecbd43571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.423715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.423629    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.423715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.423661    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkbqd\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-kube-api-access-zkbqd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.423715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.423673    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed136f9-1352-45bd-9f0b-789ecbd43571-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.423715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.423681    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.423715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.423689    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed136f9-1352-45bd-9f0b-789ecbd43571-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.771071 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.771033    5958 generic.go:356] "Generic (PLEG): container finished" podID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerID="65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb" exitCode=0
Oct 31 20:04:16.771280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.771069    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" event={"ID":"3ed136f9-1352-45bd-9f0b-789ecbd43571","Type":"ContainerDied","Data":"65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb"}
Oct 31 20:04:16.771280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.771133    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv" event={"ID":"3ed136f9-1352-45bd-9f0b-789ecbd43571","Type":"ContainerDied","Data":"5e36af7292357e322997b9c02a325796c499b70f5add01c94acfd959c43fcb34"}
Oct 31 20:04:16.771280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.771088    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"
Oct 31 20:04:16.771280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.771162    5958 scope.go:122] "RemoveContainer" containerID="65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb"
Oct 31 20:04:16.783072 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.783052    5958 scope.go:122] "RemoveContainer" containerID="65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb"
Oct 31 20:04:16.783408 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:16.783387    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb\": container with ID starting with 65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb not found: ID does not exist" containerID="65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb"
Oct 31 20:04:16.783479 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.783417    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb"} err="failed to get container status \"65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb\": rpc error: code = NotFound desc = could not find container \"65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb\": container with ID starting with 65223c9136b9c3873cd6e167407cb22963178b69ff2eb36258613044b0aa77eb not found: ID does not exist"
Oct 31 20:04:16.803781 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.803729    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"]
Oct 31 20:04:16.812757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:16.812705    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-lb2gv"]
Oct 31 20:04:17.049063 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:17.048977    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:04:18.666236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:18.666198    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" path="/var/lib/kubelet/pods/3ed136f9-1352-45bd-9f0b-789ecbd43571/volumes"
Oct 31 20:04:20.890775 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.890739    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"]
Oct 31 20:04:20.891859 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891837    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" containerName="payload-pre-processing"
Oct 31 20:04:20.891924 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891863    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" containerName="payload-pre-processing"
Oct 31 20:04:20.891924 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891914    5958 cpu_manager.go:414] "containerMap: removing container" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" containerName="payload-processing"
Oct 31 20:04:20.891924 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891921    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" containerName="payload-processing"
Oct 31 20:04:20.892026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891955    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy"
Oct 31 20:04:20.892026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891962    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy"
Oct 31 20:04:20.892026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891980    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2c686a0b-589f-4436-a395-ad20a393e79c" containerName="maas-api"
Oct 31 20:04:20.892026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.891987    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c686a0b-589f-4436-a395-ad20a393e79c" containerName="maas-api"
Oct 31 20:04:20.892026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892006    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy"
Oct 31 20:04:20.892026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892014    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy"
Oct 31 20:04:20.892226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892027    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy"
Oct 31 20:04:20.892226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892034    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy"
Oct 31 20:04:20.892226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892206    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed136f9-1352-45bd-9f0b-789ecbd43571" containerName="istio-proxy"
Oct 31 20:04:20.892226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892219    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c686a0b-589f-4436-a395-ad20a393e79c" containerName="maas-api"
Oct 31 20:04:20.892467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892229    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e374bf0e-a2e2-49aa-b9a4-b0bb77753193" containerName="istio-proxy"
Oct 31 20:04:20.892467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892241    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1880b0c3-b0c0-40f4-b21e-3885303d14da" containerName="istio-proxy"
Oct 31 20:04:20.892467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892254    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65915d4-8e42-45ab-abc6-b539629821cf" containerName="payload-pre-processing"
Oct 31 20:04:20.892467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.892270    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="07738ae8-6f04-404f-80ef-06031d966bfb" containerName="payload-processing"
Oct 31 20:04:20.898719 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.898691    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:20.906403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.906363    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"]
Oct 31 20:04:20.912752 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.912727    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"]
Oct 31 20:04:20.912932 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.912823    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:20.916649 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.916047    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f28d48-dockercfg-r4vkr\""
Oct 31 20:04:20.930302 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.930267    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"]
Oct 31 20:04:20.930469 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:20.930427    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f28d48\""
Oct 31 20:04:21.020457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.011447    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8r4\" (UniqueName: \"kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.020457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.013232    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvp6\" (UniqueName: \"kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6\") pod \"payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.020457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.014322    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.020457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.014476    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.125567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.122678    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.125567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.122790    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.125567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.122970    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8r4\" (UniqueName: \"kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.125567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.123024    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvp6\" (UniqueName: \"kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6\") pod \"payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.125567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.124383    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.128381 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.127300    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.146621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.144632    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvp6\" (UniqueName: \"kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6\") pod \"payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.147389 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.147359    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8r4\" (UniqueName: \"kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.232627 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.232595    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.248112 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.248080    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.537174 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:04:21.537142    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96842103_1def_4b3a_9321_2e4434e33a5b.slice/crio-ee8695280f0e2d8d08e6a077ded772eeefaf4e67aab3774d2791b6da333b81ab WatchSource:0}: Error finding container ee8695280f0e2d8d08e6a077ded772eeefaf4e67aab3774d2791b6da333b81ab: Status 404 returned error can't find the container with id ee8695280f0e2d8d08e6a077ded772eeefaf4e67aab3774d2791b6da333b81ab
Oct 31 20:04:21.537787 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.537765    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"]
Oct 31 20:04:21.637114 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:04:21.637086    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f562b3_c37e_4710_a571_614799454fc6.slice/crio-77c3fa073057198f7eeda1b97b3415c95d999cc2ef530e3676542ad936f08006 WatchSource:0}: Error finding container 77c3fa073057198f7eeda1b97b3415c95d999cc2ef530e3676542ad936f08006: Status 404 returned error can't find the container with id 77c3fa073057198f7eeda1b97b3415c95d999cc2ef530e3676542ad936f08006
Oct 31 20:04:21.637634 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.637610    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"]
Oct 31 20:04:21.815579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.815532    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b" event={"ID":"96842103-1def-4b3a-9321-2e4434e33a5b","Type":"ContainerStarted","Data":"a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a"}
Oct 31 20:04:21.815809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.815600    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b" event={"ID":"96842103-1def-4b3a-9321-2e4434e33a5b","Type":"ContainerStarted","Data":"ee8695280f0e2d8d08e6a077ded772eeefaf4e67aab3774d2791b6da333b81ab"}
Oct 31 20:04:21.815809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.815621    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:21.817491 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.817463    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6" event={"ID":"f0f562b3-c37e-4710-a571-614799454fc6","Type":"ContainerStarted","Data":"e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545"}
Oct 31 20:04:21.817654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.817517    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:21.817654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.817535    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6" event={"ID":"f0f562b3-c37e-4710-a571-614799454fc6","Type":"ContainerStarted","Data":"77c3fa073057198f7eeda1b97b3415c95d999cc2ef530e3676542ad936f08006"}
Oct 31 20:04:21.846511 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.846454    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b" podStartSLOduration=1.846438931 podStartE2EDuration="1.846438931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:21.842417584 +0000 UTC m=+2655.303792200" watchObservedRunningTime="2026-10-31 20:04:21.846438931 +0000 UTC m=+2655.307813546"
Oct 31 20:04:21.867555 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:21.867498    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6" podStartSLOduration=1.8674840449999999 podStartE2EDuration="1.867484045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:21.86699459 +0000 UTC m=+2655.328369205" watchObservedRunningTime="2026-10-31 20:04:21.867484045 +0000 UTC m=+2655.328858660"
Oct 31 20:04:25.376967 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:25.376926    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"]
Oct 31 20:04:25.398584 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:25.398507    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"]
Oct 31 20:04:30.378685 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:30.378642    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" containerName="payload-processing" containerID="cri-o://d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951" gracePeriod=25
Oct 31 20:04:30.399227 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:30.399182    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" containerName="payload-pre-processing" containerID="cri-o://a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae" gracePeriod=25
Oct 31 20:04:32.828352 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:32.828318    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:04:32.829987 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:32.829963    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:04:32.841892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:32.841857    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"]
Oct 31 20:04:32.842149 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:32.842114    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478" podUID="f5f1bebb-6d87-417c-88b7-4e41031f6a83" containerName="maas-api" containerID="cri-o://24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e" gracePeriod=30
Oct 31 20:04:33.399423 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.399388    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 20:04:33.488929 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.488899    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca\") pod \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") "
Oct 31 20:04:33.488929 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.488938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs\") pod \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") "
Oct 31 20:04:33.489180 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.489118    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd\" (UniqueName: \"kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd\") pod \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") "
Oct 31 20:04:33.489219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.489194    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls\") pod \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\" (UID: \"f5f1bebb-6d87-417c-88b7-4e41031f6a83\") "
Oct 31 20:04:33.489360 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.489336    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca" pod "f5f1bebb-6d87-417c-88b7-4e41031f6a83" (UID: "f5f1bebb-6d87-417c-88b7-4e41031f6a83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:33.490516 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.490487    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f1bebb-6d87-417c-88b7-4e41031f6a83-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:33.492835 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.492810    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs" pod "f5f1bebb-6d87-417c-88b7-4e41031f6a83" (UID: "f5f1bebb-6d87-417c-88b7-4e41031f6a83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:33.492952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.492833    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls" pod "f5f1bebb-6d87-417c-88b7-4e41031f6a83" (UID: "f5f1bebb-6d87-417c-88b7-4e41031f6a83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:33.492952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.492878    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd" pod "f5f1bebb-6d87-417c-88b7-4e41031f6a83" (UID: "f5f1bebb-6d87-417c-88b7-4e41031f6a83"). InnerVolumeSpecName "kube-api-access-wswkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:33.593148 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.593111    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:33.593148 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.593148    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wswkd\" (UniqueName: \"kubernetes.io/projected/f5f1bebb-6d87-417c-88b7-4e41031f6a83-kube-api-access-wswkd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:33.593363 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.593163    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f1bebb-6d87-417c-88b7-4e41031f6a83-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:33.930186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.928571    5958 generic.go:356] "Generic (PLEG): container finished" podID="f5f1bebb-6d87-417c-88b7-4e41031f6a83" containerID="24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e" exitCode=0
Oct 31 20:04:33.930186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.928717    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478" event={"ID":"f5f1bebb-6d87-417c-88b7-4e41031f6a83","Type":"ContainerDied","Data":"24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e"}
Oct 31 20:04:33.930186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.928769    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478" event={"ID":"f5f1bebb-6d87-417c-88b7-4e41031f6a83","Type":"ContainerDied","Data":"8ec4f060da2a986ad185404ab23545c69abe09c2daa32da7cb8cb2da7b52111f"}
Oct 31 20:04:33.930186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.928795    5958 scope.go:122] "RemoveContainer" containerID="24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e"
Oct 31 20:04:33.930186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.928968    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"
Oct 31 20:04:33.944486 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.944463    5958 scope.go:122] "RemoveContainer" containerID="24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e"
Oct 31 20:04:33.944906 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:33.944883    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e\": container with ID starting with 24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e not found: ID does not exist" containerID="24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e"
Oct 31 20:04:33.945042 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.944916    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e"} err="failed to get container status \"24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e\": rpc error: code = NotFound desc = could not find container \"24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e\": container with ID starting with 24db660d7acab3fcd09e8ebb85bcc0f9985dd7c2e8ee71e5b502baf0e2ba477e not found: ID does not exist"
Oct 31 20:04:33.971957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.971922    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"]
Oct 31 20:04:33.976232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:33.976199    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e618e1-b9577c87b-bw478"]
Oct 31 20:04:34.364006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:34.363912    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:04:34.364162 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:34.363915    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:04:34.665759 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:34.665728    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f1bebb-6d87-417c-88b7-4e41031f6a83" path="/var/lib/kubelet/pods/f5f1bebb-6d87-417c-88b7-4e41031f6a83/volumes"
Oct 31 20:04:36.107779 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.107746    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"]
Oct 31 20:04:36.108981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.108958    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f5f1bebb-6d87-417c-88b7-4e41031f6a83" containerName="maas-api"
Oct 31 20:04:36.108981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.108979    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f1bebb-6d87-417c-88b7-4e41031f6a83" containerName="maas-api"
Oct 31 20:04:36.109247 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.109233    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f1bebb-6d87-417c-88b7-4e41031f6a83" containerName="maas-api"
Oct 31 20:04:36.115114 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.115091    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.126344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.126313    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"]
Oct 31 20:04:36.141373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.141339    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4fw4\" (UniqueName: \"kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.141567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.141530    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.141875 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.141851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.141973 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.141900    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.161857 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.161595    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f28d48-serving-cert\""
Oct 31 20:04:36.243453 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.243407    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.243699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.243495    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.243699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.243594    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.243846 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.243831    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4fw4\" (UniqueName: \"kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.244439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.244412    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.246839 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.246813    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.246946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.246813    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.257817 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.257787    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4fw4\" (UniqueName: \"kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4\") pod \"maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.433829 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.433794    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:36.759821 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.759788    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"]
Oct 31 20:04:36.764733 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:04:36.764698    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5783b36a_c79f_47b7_a407_cbfaf48567c1.slice/crio-7880b5cc20c58b2d818354f35906ff02bacb10ed8440624090bd37d17a1f0b3d WatchSource:0}: Error finding container 7880b5cc20c58b2d818354f35906ff02bacb10ed8440624090bd37d17a1f0b3d: Status 404 returned error can't find the container with id 7880b5cc20c58b2d818354f35906ff02bacb10ed8440624090bd37d17a1f0b3d
Oct 31 20:04:36.765905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.765879    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:04:36.957241 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:36.957207    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb" event={"ID":"5783b36a-c79f-47b7-a407-cbfaf48567c1","Type":"ContainerStarted","Data":"7880b5cc20c58b2d818354f35906ff02bacb10ed8440624090bd37d17a1f0b3d"}
Oct 31 20:04:37.966828 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:37.966793    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb" event={"ID":"5783b36a-c79f-47b7-a407-cbfaf48567c1","Type":"ContainerStarted","Data":"6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002"}
Oct 31 20:04:37.967273 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:37.966869    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:37.997936 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:37.997886    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb" podStartSLOduration=1.581404147 podStartE2EDuration="1.997871444s" totalImagesPullingTime="416.467297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:37.991973215 +0000 UTC m=+2671.453347843" watchObservedRunningTime="2026-10-31 20:04:37.997871444 +0000 UTC m=+2671.459246058"
Oct 31 20:04:38.621509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:38.620575    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"]
Oct 31 20:04:38.621509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:38.620864    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk" podUID="607f4e4f-8aad-42d8-a162-f4235675ccac" containerName="maas-api" containerID="cri-o://0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97" gracePeriod=30
Oct 31 20:04:38.756642 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:38.755824    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"]
Oct 31 20:04:38.756642 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:38.756084    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf" podUID="37467c1d-a0f5-462d-a9ee-4e98ef27713f" containerName="payload-pre-processing" containerID="cri-o://3de88dd0f80b4c15cf671beea152366bbf2e66d6b1d37ad84ca2c697adc1dac9" gracePeriod=30
Oct 31 20:04:39.105081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.103620    5958 generic.go:356] "Generic (PLEG): container finished" podID="37467c1d-a0f5-462d-a9ee-4e98ef27713f" containerID="3de88dd0f80b4c15cf671beea152366bbf2e66d6b1d37ad84ca2c697adc1dac9" exitCode=0
Oct 31 20:04:39.105081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.103363    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:04:39.105081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.103807    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf" event={"ID":"37467c1d-a0f5-462d-a9ee-4e98ef27713f","Type":"ContainerDied","Data":"3de88dd0f80b4c15cf671beea152366bbf2e66d6b1d37ad84ca2c697adc1dac9"}
Oct 31 20:04:39.117217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.116619    5958 generic.go:356] "Generic (PLEG): container finished" podID="607f4e4f-8aad-42d8-a162-f4235675ccac" containerID="0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97" exitCode=0
Oct 31 20:04:39.117217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.117015    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk" event={"ID":"607f4e4f-8aad-42d8-a162-f4235675ccac","Type":"ContainerDied","Data":"0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97"}
Oct 31 20:04:39.117217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.117060    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk" event={"ID":"607f4e4f-8aad-42d8-a162-f4235675ccac","Type":"ContainerDied","Data":"6317c54a586bd6dac98178338132277771e81180486a0655aaf955cc60562c3b"}
Oct 31 20:04:39.117217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.117089    5958 scope.go:122] "RemoveContainer" containerID="0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97"
Oct 31 20:04:39.143693 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.143080    5958 scope.go:122] "RemoveContainer" containerID="0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97"
Oct 31 20:04:39.143995 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:39.143963    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97\": container with ID starting with 0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97 not found: ID does not exist" containerID="0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97"
Oct 31 20:04:39.144124 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.144000    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97"} err="failed to get container status \"0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97\": rpc error: code = NotFound desc = could not find container \"0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97\": container with ID starting with 0b3a8d7dd32e865b8483c81c50bfa42184a8e4f6e19ebe843e236d8340b95c97 not found: ID does not exist"
Oct 31 20:04:39.221125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.221088    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs\") pod \"607f4e4f-8aad-42d8-a162-f4235675ccac\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") "
Oct 31 20:04:39.221293 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.221219    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca\") pod \"607f4e4f-8aad-42d8-a162-f4235675ccac\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") "
Oct 31 20:04:39.221383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.221324    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls\") pod \"607f4e4f-8aad-42d8-a162-f4235675ccac\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") "
Oct 31 20:04:39.221482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.221464    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz\" (UniqueName: \"kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz\") pod \"607f4e4f-8aad-42d8-a162-f4235675ccac\" (UID: \"607f4e4f-8aad-42d8-a162-f4235675ccac\") "
Oct 31 20:04:39.223680 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.223648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca" pod "607f4e4f-8aad-42d8-a162-f4235675ccac" (UID: "607f4e4f-8aad-42d8-a162-f4235675ccac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:39.224265 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.224242    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:04:39.225370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.225294    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs" pod "607f4e4f-8aad-42d8-a162-f4235675ccac" (UID: "607f4e4f-8aad-42d8-a162-f4235675ccac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:39.225862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.225833    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz" pod "607f4e4f-8aad-42d8-a162-f4235675ccac" (UID: "607f4e4f-8aad-42d8-a162-f4235675ccac"). InnerVolumeSpecName "kube-api-access-9zdpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:39.226894 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.226870    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls" pod "607f4e4f-8aad-42d8-a162-f4235675ccac" (UID: "607f4e4f-8aad-42d8-a162-f4235675ccac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:39.327161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.327117    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume\") pod \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") "
Oct 31 20:04:39.339825 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.339790    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk\" (UniqueName: \"kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk\") pod \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\" (UID: \"37467c1d-a0f5-462d-a9ee-4e98ef27713f\") "
Oct 31 20:04:39.344324 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.344285    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zdpz\" (UniqueName: \"kubernetes.io/projected/607f4e4f-8aad-42d8-a162-f4235675ccac-kube-api-access-9zdpz\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.344621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.344600    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.344767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.344754    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/607f4e4f-8aad-42d8-a162-f4235675ccac-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.344870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.344859    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/607f4e4f-8aad-42d8-a162-f4235675ccac-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.344957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.338343    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume" pod "37467c1d-a0f5-462d-a9ee-4e98ef27713f" (UID: "37467c1d-a0f5-462d-a9ee-4e98ef27713f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:39.351161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.349681    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk" pod "37467c1d-a0f5-462d-a9ee-4e98ef27713f" (UID: "37467c1d-a0f5-462d-a9ee-4e98ef27713f"). InnerVolumeSpecName "kube-api-access-8bbwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:39.446497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.446414    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37467c1d-a0f5-462d-a9ee-4e98ef27713f-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.446497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:39.446444    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbwk\" (UniqueName: \"kubernetes.io/projected/37467c1d-a0f5-462d-a9ee-4e98ef27713f-kube-api-access-8bbwk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:40.126472 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.126443    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"
Oct 31 20:04:40.126997 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.126482    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf" event={"ID":"37467c1d-a0f5-462d-a9ee-4e98ef27713f","Type":"ContainerDied","Data":"13cc37b853529519b835940f4f67e90f1c9cb438f75c8848dab3e222fc04905d"}
Oct 31 20:04:40.126997 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.126577    5958 scope.go:122] "RemoveContainer" containerID="3de88dd0f80b4c15cf671beea152366bbf2e66d6b1d37ad84ca2c697adc1dac9"
Oct 31 20:04:40.127948 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.127929    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"
Oct 31 20:04:40.163757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.163720    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"]
Oct 31 20:04:40.167903 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.167874    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a84fcb-8cbf7d77-s5qlk"]
Oct 31 20:04:40.186741 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.186674    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"]
Oct 31 20:04:40.196105 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.196060    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a84fcb-db98dd6f9-hmqdf"]
Oct 31 20:04:40.682043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.682008    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37467c1d-a0f5-462d-a9ee-4e98ef27713f" path="/var/lib/kubelet/pods/37467c1d-a0f5-462d-a9ee-4e98ef27713f/volumes"
Oct 31 20:04:40.682582 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:40.682538    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607f4e4f-8aad-42d8-a162-f4235675ccac" path="/var/lib/kubelet/pods/607f4e4f-8aad-42d8-a162-f4235675ccac/volumes"
Oct 31 20:04:43.561606 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:43.560926    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"]
Oct 31 20:04:43.561606 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:43.561197    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" containerID="cri-o://fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d" gracePeriod=30
Oct 31 20:04:43.566358 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:43.566289    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"]
Oct 31 20:04:43.566677 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:43.566600    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" containerID="cri-o://9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1" gracePeriod=30
Oct 31 20:04:44.122451 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:44.122420    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:04:44.363413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:44.363369    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:04:44.363646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:44.363369    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:04:45.294692 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:45.294647    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:45.295111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:45.294735    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:45.950502 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:45.950461    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:45.950712 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:45.950530    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:47.047975 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:47.047935    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:04:48.879671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.879644    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 20:04:48.939363 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.939337    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 20:04:48.952019 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.951987    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952249 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952091    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952249 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952152    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952249 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952212    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952332    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.952433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952285    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952496    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952526    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952566    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952533    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.952808 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952600    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data\") pod \"9193f73f-a5b0-4033-ac97-94c990e4019c\" (UID: \"9193f73f-a5b0-4033-ac97-94c990e4019c\") "
Oct 31 20:04:48.952944 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.952899    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.953123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.953103    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.953377 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.953354    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:48.953498 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.953478    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.953626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.953507    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.953626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.953524    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.953626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.953537    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.955204 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.955173    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "kube-api-access-4c5pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:48.955329 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.955314    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:48.955477 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.955458    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:48.955577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:48.955524    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy" pod "9193f73f-a5b0-4033-ac97-94c990e4019c" (UID: "9193f73f-a5b0-4033-ac97-94c990e4019c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056195    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056255    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056298    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056781    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056843    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056948    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.056998    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.057036    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.057080    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data\") pod \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\" (UID: \"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3\") "
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.058208    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.058251    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5pk\" (UniqueName: \"kubernetes.io/projected/9193f73f-a5b0-4033-ac97-94c990e4019c-kube-api-access-4c5pk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.058274    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.058294    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9193f73f-a5b0-4033-ac97-94c990e4019c-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.060315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.058310    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9193f73f-a5b0-4033-ac97-94c990e4019c-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.071098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.066762    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.071098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.067227    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:49.071098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.067564    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.075156 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.074052    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.075156 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.074579    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.087007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.086883    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "kube-api-access-946cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.087744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.087681    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.088698 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.088636    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:49.089136 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.089097    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy" pod "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" (UID: "e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.160333 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160289    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160333 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160332    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160348    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160363    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160379    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160393    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160407    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160423    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.160567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.160439    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946cm\" (UniqueName: \"kubernetes.io/projected/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3-kube-api-access-946cm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.209115 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.208992    5958 generic.go:356] "Generic (PLEG): container finished" podID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerID="fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d" exitCode=0
Oct 31 20:04:49.209115 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.209048    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"
Oct 31 20:04:49.209443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.209169    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" event={"ID":"9193f73f-a5b0-4033-ac97-94c990e4019c","Type":"ContainerDied","Data":"fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d"}
Oct 31 20:04:49.209443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.209212    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9" event={"ID":"9193f73f-a5b0-4033-ac97-94c990e4019c","Type":"ContainerDied","Data":"3457b0e43b716ee924cce9a7547d8ccfc39bfdb4ae1249c2818ef128bbffdcc2"}
Oct 31 20:04:49.209443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.209273    5958 scope.go:122] "RemoveContainer" containerID="fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d"
Oct 31 20:04:49.211707 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.211673    5958 generic.go:356] "Generic (PLEG): container finished" podID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerID="9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1" exitCode=0
Oct 31 20:04:49.212638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.211732    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"
Oct 31 20:04:49.212638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.211713    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" event={"ID":"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3","Type":"ContainerDied","Data":"9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1"}
Oct 31 20:04:49.212638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.211801    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2" event={"ID":"e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3","Type":"ContainerDied","Data":"54bc20850c7707232b799c37ddfa7020a7999b7c9833b0edd573e9ea3e1af57d"}
Oct 31 20:04:49.223803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.223783    5958 scope.go:122] "RemoveContainer" containerID="fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d"
Oct 31 20:04:49.224135 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:49.224106    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d\": container with ID starting with fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d not found: ID does not exist" containerID="fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d"
Oct 31 20:04:49.224218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.224136    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d"} err="failed to get container status \"fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d\": rpc error: code = NotFound desc = could not find container \"fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d\": container with ID starting with fa1c5719660124aacad4ec480da2b32a3d5e1db13270c628c1cb3bbde5d3614d not found: ID does not exist"
Oct 31 20:04:49.224218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.224152    5958 scope.go:122] "RemoveContainer" containerID="9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1"
Oct 31 20:04:49.242984 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.242955    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"]
Oct 31 20:04:49.245098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.245074    5958 scope.go:122] "RemoveContainer" containerID="9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1"
Oct 31 20:04:49.245423 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:49.245401    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1\": container with ID starting with 9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1 not found: ID does not exist" containerID="9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1"
Oct 31 20:04:49.245472 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.245432    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1"} err="failed to get container status \"9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1\": rpc error: code = NotFound desc = could not find container \"9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1\": container with ID starting with 9ba31726bbc7604e0dea9def665641e059cc61ce95d9e58e897da416149435a1 not found: ID does not exist"
Oct 31 20:04:49.246872 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.246848    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-kx8c9"]
Oct 31 20:04:49.257767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.257738    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"]
Oct 31 20:04:49.266588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.266538    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-bxld2"]
Oct 31 20:04:49.613572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:49.613520    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:04:50.678104 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.678067    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" path="/var/lib/kubelet/pods/9193f73f-a5b0-4033-ac97-94c990e4019c/volumes"
Oct 31 20:04:50.679639 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.679606    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" path="/var/lib/kubelet/pods/e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3/volumes"
Oct 31 20:04:50.835827 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.835788    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 20:04:50.950627 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.950453    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8\" (UniqueName: \"kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8\") pod \"3d461c42-cd97-42d7-9098-b4911126ce01\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") "
Oct 31 20:04:50.951264 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.951235    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume\") pod \"3d461c42-cd97-42d7-9098-b4911126ce01\" (UID: \"3d461c42-cd97-42d7-9098-b4911126ce01\") "
Oct 31 20:04:50.952855 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.952828    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume" pod "3d461c42-cd97-42d7-9098-b4911126ce01" (UID: "3d461c42-cd97-42d7-9098-b4911126ce01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:50.971344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:50.971299    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8" pod "3d461c42-cd97-42d7-9098-b4911126ce01" (UID: "3d461c42-cd97-42d7-9098-b4911126ce01"). InnerVolumeSpecName "kube-api-access-k4wl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:51.005161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.005127    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 20:04:51.058949 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.058894    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d461c42-cd97-42d7-9098-b4911126ce01-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:51.059266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.059143    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4wl8\" (UniqueName: \"kubernetes.io/projected/3d461c42-cd97-42d7-9098-b4911126ce01-kube-api-access-k4wl8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:51.160745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.160706    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98\" (UniqueName: \"kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98\") pod \"b66eba36-4f37-41dd-86e0-35987e97506d\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") "
Oct 31 20:04:51.161127 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.161103    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume\") pod \"b66eba36-4f37-41dd-86e0-35987e97506d\" (UID: \"b66eba36-4f37-41dd-86e0-35987e97506d\") "
Oct 31 20:04:51.161480 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.161451    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume" pod "b66eba36-4f37-41dd-86e0-35987e97506d" (UID: "b66eba36-4f37-41dd-86e0-35987e97506d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:51.161687 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.161672    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66eba36-4f37-41dd-86e0-35987e97506d-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:51.162884 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.162867    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98" pod "b66eba36-4f37-41dd-86e0-35987e97506d" (UID: "b66eba36-4f37-41dd-86e0-35987e97506d"). InnerVolumeSpecName "kube-api-access-r4d98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:51.263387 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.263300    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4d98\" (UniqueName: \"kubernetes.io/projected/b66eba36-4f37-41dd-86e0-35987e97506d-kube-api-access-r4d98\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:51.274939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.274906    5958 generic.go:356] "Generic (PLEG): container finished" podID="3d461c42-cd97-42d7-9098-b4911126ce01" containerID="d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951" exitCode=0
Oct 31 20:04:51.275130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.274953    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"
Oct 31 20:04:51.275130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.274982    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" event={"ID":"3d461c42-cd97-42d7-9098-b4911126ce01","Type":"ContainerDied","Data":"d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951"}
Oct 31 20:04:51.275130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.275044    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4" event={"ID":"3d461c42-cd97-42d7-9098-b4911126ce01","Type":"ContainerDied","Data":"2aefb9d6120340658f8cda82f107256ffd2a94983fcba701567a86b8c350bddc"}
Oct 31 20:04:51.275130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.275071    5958 scope.go:122] "RemoveContainer" containerID="d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951"
Oct 31 20:04:51.276779 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.276727    5958 generic.go:356] "Generic (PLEG): container finished" podID="b66eba36-4f37-41dd-86e0-35987e97506d" containerID="a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae" exitCode=0
Oct 31 20:04:51.276779 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.276762    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" event={"ID":"b66eba36-4f37-41dd-86e0-35987e97506d","Type":"ContainerDied","Data":"a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae"}
Oct 31 20:04:51.276951 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.276819    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd" event={"ID":"b66eba36-4f37-41dd-86e0-35987e97506d","Type":"ContainerDied","Data":"d221c7e3c77ff8bea32f361f866232cbecdaa5191c34c500f3befee1d7d4141f"}
Oct 31 20:04:51.276951 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.276774    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"
Oct 31 20:04:51.286943 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.286924    5958 scope.go:122] "RemoveContainer" containerID="d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951"
Oct 31 20:04:51.287243 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:51.287224    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951\": container with ID starting with d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951 not found: ID does not exist" containerID="d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951"
Oct 31 20:04:51.287313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.287254    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951"} err="failed to get container status \"d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951\": rpc error: code = NotFound desc = could not find container \"d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951\": container with ID starting with d7eb83e4543dc65e447cf5bfd85709d4b69146393146ec2020ce45df41ebc951 not found: ID does not exist"
Oct 31 20:04:51.287313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.287271    5958 scope.go:122] "RemoveContainer" containerID="a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae"
Oct 31 20:04:51.298385 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.298366    5958 scope.go:122] "RemoveContainer" containerID="a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae"
Oct 31 20:04:51.298756 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:04:51.298726    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae\": container with ID starting with a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae not found: ID does not exist" containerID="a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae"
Oct 31 20:04:51.298847 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.298756    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae"} err="failed to get container status \"a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae\": rpc error: code = NotFound desc = could not find container \"a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae\": container with ID starting with a2046b073169270fc99a8e05b8db28894e2afb0e270a04f052f5db7430f79bae not found: ID does not exist"
Oct 31 20:04:51.343355 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.343323    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"]
Oct 31 20:04:51.362974 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.362943    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-e618e1-7cddf5f95c-xd7c4"]
Oct 31 20:04:51.428213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.428180    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"]
Oct 31 20:04:51.442424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:51.442393    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-e618e1-5f6cdf8759-qwvgd"]
Oct 31 20:04:52.665977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:52.665940    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" path="/var/lib/kubelet/pods/3d461c42-cd97-42d7-9098-b4911126ce01/volumes"
Oct 31 20:04:52.666445 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:52.666430    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" path="/var/lib/kubelet/pods/b66eba36-4f37-41dd-86e0-35987e97506d/volumes"
Oct 31 20:04:53.036072 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:53.035989    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"]
Oct 31 20:04:53.055301 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:53.055264    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"]
Oct 31 20:04:58.037752 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:58.037705    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" podUID="5602cf8f-d9ef-412b-8901-91285525c030" containerName="payload-processing" containerID="cri-o://e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3" gracePeriod=25
Oct 31 20:04:58.055715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:58.055675    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerName="payload-pre-processing" containerID="cri-o://cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e" gracePeriod=25
Oct 31 20:04:58.599897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:04:58.599846    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"]
Oct 31 20:05:00.288016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:00.287980    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"]
Oct 31 20:05:00.303069 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:00.303031    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"]
Oct 31 20:05:03.600503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:03.600455    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerName="payload-pre-processing" containerID="cri-o://b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561" gracePeriod=25
Oct 31 20:05:04.007493 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:04.007452    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" podUID="5602cf8f-d9ef-412b-8901-91285525c030" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:05.073572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.072754    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:05.076638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.076595    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"]
Oct 31 20:05:05.089344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.089303    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbcbf96-5q6g7"]
Oct 31 20:05:05.265009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.264972    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"]
Oct 31 20:05:05.265418 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.265387    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" containerID="cri-o://df685ad9701762aea6e0ed9302038d29e11c0fe95b2d7823e89bf0be6e1f9d94" gracePeriod=30
Oct 31 20:05:05.288526 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.288480    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6" podUID="f0f562b3-c37e-4710-a571-614799454fc6" containerName="payload-processing" containerID="cri-o://e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545" gracePeriod=25
Oct 31 20:05:05.303587 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.303531    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b" podUID="96842103-1def-4b3a-9321-2e4434e33a5b" containerName="payload-pre-processing" containerID="cri-o://a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a" gracePeriod=25
Oct 31 20:05:05.666641 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.666296    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:05:05.739539 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.739509    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:05:05.800340 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.800300    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:05.800587 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.800398    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:05.874890 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.874856    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4\" (UniqueName: \"kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4\") pod \"96842103-1def-4b3a-9321-2e4434e33a5b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") "
Oct 31 20:05:05.875102 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.874957    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6\" (UniqueName: \"kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6\") pod \"f0f562b3-c37e-4710-a571-614799454fc6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") "
Oct 31 20:05:05.875157 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.875114    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume\") pod \"f0f562b3-c37e-4710-a571-614799454fc6\" (UID: \"f0f562b3-c37e-4710-a571-614799454fc6\") "
Oct 31 20:05:05.875314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.875292    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume\") pod \"96842103-1def-4b3a-9321-2e4434e33a5b\" (UID: \"96842103-1def-4b3a-9321-2e4434e33a5b\") "
Oct 31 20:05:05.875688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.875568    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume" pod "f0f562b3-c37e-4710-a571-614799454fc6" (UID: "f0f562b3-c37e-4710-a571-614799454fc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:05.876601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.875972    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume" pod "96842103-1def-4b3a-9321-2e4434e33a5b" (UID: "96842103-1def-4b3a-9321-2e4434e33a5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:05.876601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.876296    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96842103-1def-4b3a-9321-2e4434e33a5b-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:05.876601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.876334    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f562b3-c37e-4710-a571-614799454fc6-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:05.877233 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.877214    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6" pod "f0f562b3-c37e-4710-a571-614799454fc6" (UID: "f0f562b3-c37e-4710-a571-614799454fc6"). InnerVolumeSpecName "kube-api-access-wlvp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:05.877491 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.877471    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4" pod "96842103-1def-4b3a-9321-2e4434e33a5b" (UID: "96842103-1def-4b3a-9321-2e4434e33a5b"). InnerVolumeSpecName "kube-api-access-kh8r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:05.977655 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.977525    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh8r4\" (UniqueName: \"kubernetes.io/projected/96842103-1def-4b3a-9321-2e4434e33a5b-kube-api-access-kh8r4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:05.977655 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:05.977609    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvp6\" (UniqueName: \"kubernetes.io/projected/f0f562b3-c37e-4710-a571-614799454fc6-kube-api-access-wlvp6\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:06.446902 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.446864    5958 generic.go:356] "Generic (PLEG): container finished" podID="f0f562b3-c37e-4710-a571-614799454fc6" containerID="e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545" exitCode=0
Oct 31 20:05:06.447356 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.446915    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"
Oct 31 20:05:06.447356 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.446943    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6" event={"ID":"f0f562b3-c37e-4710-a571-614799454fc6","Type":"ContainerDied","Data":"e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545"}
Oct 31 20:05:06.447356 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.446995    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6" event={"ID":"f0f562b3-c37e-4710-a571-614799454fc6","Type":"ContainerDied","Data":"77c3fa073057198f7eeda1b97b3415c95d999cc2ef530e3676542ad936f08006"}
Oct 31 20:05:06.447356 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.447020    5958 scope.go:122] "RemoveContainer" containerID="e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545"
Oct 31 20:05:06.448722 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.448685    5958 generic.go:356] "Generic (PLEG): container finished" podID="96842103-1def-4b3a-9321-2e4434e33a5b" containerID="a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a" exitCode=0
Oct 31 20:05:06.448849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.448827    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b" event={"ID":"96842103-1def-4b3a-9321-2e4434e33a5b","Type":"ContainerDied","Data":"a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a"}
Oct 31 20:05:06.448974 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.448888    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b" event={"ID":"96842103-1def-4b3a-9321-2e4434e33a5b","Type":"ContainerDied","Data":"ee8695280f0e2d8d08e6a077ded772eeefaf4e67aab3774d2791b6da333b81ab"}
Oct 31 20:05:06.448974 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.448962    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"
Oct 31 20:05:06.460834 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.460796    5958 scope.go:122] "RemoveContainer" containerID="e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545"
Oct 31 20:05:06.461214 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:06.461185    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545\": container with ID starting with e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545 not found: ID does not exist" containerID="e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545"
Oct 31 20:05:06.461315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.461216    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545"} err="failed to get container status \"e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545\": rpc error: code = NotFound desc = could not find container \"e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545\": container with ID starting with e8cd4e27efdebe34b896c4f5f5021e68f3fe93dd062743367fa2d3b185d13545 not found: ID does not exist"
Oct 31 20:05:06.461315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.461238    5958 scope.go:122] "RemoveContainer" containerID="a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a"
Oct 31 20:05:06.472986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.472964    5958 scope.go:122] "RemoveContainer" containerID="a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a"
Oct 31 20:05:06.473394 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:06.473362    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a\": container with ID starting with a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a not found: ID does not exist" containerID="a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a"
Oct 31 20:05:06.473532 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.473394    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a"} err="failed to get container status \"a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a\": rpc error: code = NotFound desc = could not find container \"a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a\": container with ID starting with a913fe5825a523e1d7d01cdbf8e9211cd5d8fae1caf4886f520b2f192a924f7a not found: ID does not exist"
Oct 31 20:05:06.479584 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.479539    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"]
Oct 31 20:05:06.483641 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.483613    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-7745df5fc-xzcb6"]
Oct 31 20:05:06.497784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.497749    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"]
Oct 31 20:05:06.503106 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.503079    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-74946ff75-6xd4b"]
Oct 31 20:05:06.646601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.646293    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"]
Oct 31 20:05:06.646601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.646575    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x" podUID="c04b6288-9859-4958-9bc1-4104c779a292" containerName="maas-api" containerID="cri-o://f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64" gracePeriod=30
Oct 31 20:05:06.671877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.671845    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fae7b9e-3622-4f2b-8126-c50b2b986187" path="/var/lib/kubelet/pods/5fae7b9e-3622-4f2b-8126-c50b2b986187/volumes"
Oct 31 20:05:06.672654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.672562    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96842103-1def-4b3a-9321-2e4434e33a5b" path="/var/lib/kubelet/pods/96842103-1def-4b3a-9321-2e4434e33a5b/volumes"
Oct 31 20:05:06.673084 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:06.673066    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f562b3-c37e-4710-a571-614799454fc6" path="/var/lib/kubelet/pods/f0f562b3-c37e-4710-a571-614799454fc6/volumes"
Oct 31 20:05:07.067635 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.067608    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 20:05:07.204499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.204467    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs\") pod \"c04b6288-9859-4958-9bc1-4104c779a292\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") "
Oct 31 20:05:07.204721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.204538    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls\") pod \"c04b6288-9859-4958-9bc1-4104c779a292\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") "
Oct 31 20:05:07.204721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.204654    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv\" (UniqueName: \"kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv\") pod \"c04b6288-9859-4958-9bc1-4104c779a292\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") "
Oct 31 20:05:07.204851 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.204800    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca\") pod \"c04b6288-9859-4958-9bc1-4104c779a292\" (UID: \"c04b6288-9859-4958-9bc1-4104c779a292\") "
Oct 31 20:05:07.205308 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.205277    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca" pod "c04b6288-9859-4958-9bc1-4104c779a292" (UID: "c04b6288-9859-4958-9bc1-4104c779a292"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:07.207005 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.206979    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs" pod "c04b6288-9859-4958-9bc1-4104c779a292" (UID: "c04b6288-9859-4958-9bc1-4104c779a292"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:07.207220 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.207201    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls" pod "c04b6288-9859-4958-9bc1-4104c779a292" (UID: "c04b6288-9859-4958-9bc1-4104c779a292"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:07.207332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.207310    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv" pod "c04b6288-9859-4958-9bc1-4104c779a292" (UID: "c04b6288-9859-4958-9bc1-4104c779a292"). InnerVolumeSpecName "kube-api-access-pxcwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:07.306573 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.306468    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:07.306573 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.306509    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxcwv\" (UniqueName: \"kubernetes.io/projected/c04b6288-9859-4958-9bc1-4104c779a292-kube-api-access-pxcwv\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:07.306573 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.306521    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04b6288-9859-4958-9bc1-4104c779a292-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:07.306573 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.306531    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04b6288-9859-4958-9bc1-4104c779a292-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:07.458420 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.458383    5958 generic.go:356] "Generic (PLEG): container finished" podID="c04b6288-9859-4958-9bc1-4104c779a292" containerID="f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64" exitCode=0
Oct 31 20:05:07.458942 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.458437    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"
Oct 31 20:05:07.458942 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.458461    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x" event={"ID":"c04b6288-9859-4958-9bc1-4104c779a292","Type":"ContainerDied","Data":"f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64"}
Oct 31 20:05:07.458942 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.458566    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x" event={"ID":"c04b6288-9859-4958-9bc1-4104c779a292","Type":"ContainerDied","Data":"fe868af85208e9edc824c05e7d3d44f76c572e72581944248aae97cea5fbb995"}
Oct 31 20:05:07.458942 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.458599    5958 scope.go:122] "RemoveContainer" containerID="f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64"
Oct 31 20:05:07.472432 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.472409    5958 scope.go:122] "RemoveContainer" containerID="f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64"
Oct 31 20:05:07.472777 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:07.472756    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64\": container with ID starting with f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64 not found: ID does not exist" containerID="f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64"
Oct 31 20:05:07.472850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.472801    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64"} err="failed to get container status \"f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64\": rpc error: code = NotFound desc = could not find container \"f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64\": container with ID starting with f23dfe3f14660c93566c2c69ebd99bb2c8c2dd7e398ae7bc1f22c9f91132aa64 not found: ID does not exist"
Oct 31 20:05:07.492581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.492535    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"]
Oct 31 20:05:07.512783 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:07.512749    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a891ce-5d8fc5b785-pwr7x"]
Oct 31 20:05:08.667225 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:08.667181    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04b6288-9859-4958-9bc1-4104c779a292" path="/var/lib/kubelet/pods/c04b6288-9859-4958-9bc1-4104c779a292/volumes"
Oct 31 20:05:08.829593 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:08.829526    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"]
Oct 31 20:05:08.829853 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:08.829804    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk" podUID="40045daf-6b06-40e0-90ee-aac67220d7b7" containerName="maas-api" containerID="cri-o://709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af" gracePeriod=30
Oct 31 20:05:09.133744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.133715    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 20:05:09.134527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.134498    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs\") pod \"40045daf-6b06-40e0-90ee-aac67220d7b7\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") "
Oct 31 20:05:09.135058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.134941    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca\") pod \"40045daf-6b06-40e0-90ee-aac67220d7b7\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") "
Oct 31 20:05:09.135058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.135043    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr\" (UniqueName: \"kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr\") pod \"40045daf-6b06-40e0-90ee-aac67220d7b7\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") "
Oct 31 20:05:09.137687 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.135155    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls\") pod \"40045daf-6b06-40e0-90ee-aac67220d7b7\" (UID: \"40045daf-6b06-40e0-90ee-aac67220d7b7\") "
Oct 31 20:05:09.137687 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.135513    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca" pod "40045daf-6b06-40e0-90ee-aac67220d7b7" (UID: "40045daf-6b06-40e0-90ee-aac67220d7b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:09.137687 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.135858    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40045daf-6b06-40e0-90ee-aac67220d7b7-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:09.138781 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.138756    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs" pod "40045daf-6b06-40e0-90ee-aac67220d7b7" (UID: "40045daf-6b06-40e0-90ee-aac67220d7b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:09.139787 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.139759    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls" pod "40045daf-6b06-40e0-90ee-aac67220d7b7" (UID: "40045daf-6b06-40e0-90ee-aac67220d7b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:09.140687 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.140538    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr" pod "40045daf-6b06-40e0-90ee-aac67220d7b7" (UID: "40045daf-6b06-40e0-90ee-aac67220d7b7"). InnerVolumeSpecName "kube-api-access-pgtpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:09.244171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.244069    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtpr\" (UniqueName: \"kubernetes.io/projected/40045daf-6b06-40e0-90ee-aac67220d7b7-kube-api-access-pgtpr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:09.244171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.244110    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:09.244171 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.244126    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40045daf-6b06-40e0-90ee-aac67220d7b7-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:09.481315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.481279    5958 generic.go:356] "Generic (PLEG): container finished" podID="40045daf-6b06-40e0-90ee-aac67220d7b7" containerID="709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af" exitCode=0
Oct 31 20:05:09.481474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.481327    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"
Oct 31 20:05:09.481474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.481365    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk" event={"ID":"40045daf-6b06-40e0-90ee-aac67220d7b7","Type":"ContainerDied","Data":"709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af"}
Oct 31 20:05:09.481474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.481417    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk" event={"ID":"40045daf-6b06-40e0-90ee-aac67220d7b7","Type":"ContainerDied","Data":"5425d6b18981505554acc8dc74063fe555a0c66f7c58584a422f84ab3e020b34"}
Oct 31 20:05:09.481474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.481440    5958 scope.go:122] "RemoveContainer" containerID="709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af"
Oct 31 20:05:09.494775 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.494749    5958 scope.go:122] "RemoveContainer" containerID="709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af"
Oct 31 20:05:09.495113 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:09.495094    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af\": container with ID starting with 709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af not found: ID does not exist" containerID="709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af"
Oct 31 20:05:09.495208 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.495120    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af"} err="failed to get container status \"709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af\": rpc error: code = NotFound desc = could not find container \"709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af\": container with ID starting with 709a0deaa451101d3bc7f6beab9f66406cf264681fe599ecb73bc0d0f1bf35af not found: ID does not exist"
Oct 31 20:05:09.509004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.508966    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"]
Oct 31 20:05:09.516641 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:09.516608    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4bac5a-55cb76869d-bbbvk"]
Oct 31 20:05:10.499292 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.499243    5958 generic.go:356] "Generic (PLEG): container finished" podID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerID="df685ad9701762aea6e0ed9302038d29e11c0fe95b2d7823e89bf0be6e1f9d94" exitCode=0
Oct 31 20:05:10.499813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.499355    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" event={"ID":"c9284c37-4d8c-4e02-985f-b63923457bc2","Type":"ContainerDied","Data":"df685ad9701762aea6e0ed9302038d29e11c0fe95b2d7823e89bf0be6e1f9d94"}
Oct 31 20:05:10.563652 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.563582    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 20:05:10.568182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568153    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568200    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568220    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568260    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568281    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568327    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568396    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568476    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568494    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.568676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568522    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.568676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568535    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token\") pod \"c9284c37-4d8c-4e02-985f-b63923457bc2\" (UID: \"c9284c37-4d8c-4e02-985f-b63923457bc2\") "
Oct 31 20:05:10.568939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568725    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.568939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568740    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.568939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.568837    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:10.569198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.569168    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.569260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.569197    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.569260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.569213    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9284c37-4d8c-4e02-985f-b63923457bc2-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.569260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.569229    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.569260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.569243    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.570712 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.570685    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "kube-api-access-ck528". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:10.570835 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.570763    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:10.570899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.570831    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.570899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.570843    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo" pod "c9284c37-4d8c-4e02-985f-b63923457bc2" (UID: "c9284c37-4d8c-4e02-985f-b63923457bc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:10.672670 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.672609    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.672670 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.672652    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.672670 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.672674    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.673047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.672688    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck528\" (UniqueName: \"kubernetes.io/projected/c9284c37-4d8c-4e02-985f-b63923457bc2-kube-api-access-ck528\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.682342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.682291    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40045daf-6b06-40e0-90ee-aac67220d7b7" path="/var/lib/kubelet/pods/40045daf-6b06-40e0-90ee-aac67220d7b7/volumes"
Oct 31 20:05:10.740182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.740142    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"]
Oct 31 20:05:10.974276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:10.974234    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:11.511456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:11.510989    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"
Oct 31 20:05:11.511456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:11.511191    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8" event={"ID":"c9284c37-4d8c-4e02-985f-b63923457bc2","Type":"ContainerDied","Data":"c9d6d1112c7146eaea8b01a6ea3a26b49423b2dd197e2701ca896829556bb6da"}
Oct 31 20:05:11.511456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:11.511255    5958 scope.go:122] "RemoveContainer" containerID="df685ad9701762aea6e0ed9302038d29e11c0fe95b2d7823e89bf0be6e1f9d94"
Oct 31 20:05:11.538436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:11.538405    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"]
Oct 31 20:05:11.548250 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:11.548221    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-mlkk8"]
Oct 31 20:05:12.666015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:12.665983    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" path="/var/lib/kubelet/pods/c9284c37-4d8c-4e02-985f-b63923457bc2/volumes"
Oct 31 20:05:14.007846 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:14.007803    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" podUID="5602cf8f-d9ef-412b-8901-91285525c030" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:15.017063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:15.017023    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:16.050518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:16.050464    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"]
Oct 31 20:05:16.070084 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:16.070051    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"]
Oct 31 20:05:16.150732 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:16.150638    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"]
Oct 31 20:05:16.150968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:16.150881    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy" containerID="cri-o://e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c" gracePeriod=30
Oct 31 20:05:17.046627 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:17.046586    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:05:18.413182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.413155    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 20:05:18.416902 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.416879    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 20:05:18.507070 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.507038    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume\") pod \"5602cf8f-d9ef-412b-8901-91285525c030\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") "
Oct 31 20:05:18.507277 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.507115    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n\" (UniqueName: \"kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n\") pod \"5602cf8f-d9ef-412b-8901-91285525c030\" (UID: \"5602cf8f-d9ef-412b-8901-91285525c030\") "
Oct 31 20:05:18.507991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.507393    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume\") pod \"4347d253-53c6-4a1b-80a1-83a92f0202d0\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") "
Oct 31 20:05:18.507991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.507462    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume" pod "5602cf8f-d9ef-412b-8901-91285525c030" (UID: "5602cf8f-d9ef-412b-8901-91285525c030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:18.507991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.507497    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg\" (UniqueName: \"kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg\") pod \"4347d253-53c6-4a1b-80a1-83a92f0202d0\" (UID: \"4347d253-53c6-4a1b-80a1-83a92f0202d0\") "
Oct 31 20:05:18.507991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.507939    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume" pod "4347d253-53c6-4a1b-80a1-83a92f0202d0" (UID: "4347d253-53c6-4a1b-80a1-83a92f0202d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:18.508311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.508199    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5602cf8f-d9ef-412b-8901-91285525c030-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:18.508311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.508219    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4347d253-53c6-4a1b-80a1-83a92f0202d0-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:18.510168 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.510140    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n" pod "5602cf8f-d9ef-412b-8901-91285525c030" (UID: "5602cf8f-d9ef-412b-8901-91285525c030"). InnerVolumeSpecName "kube-api-access-k9c2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:18.510777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.510747    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg" pod "4347d253-53c6-4a1b-80a1-83a92f0202d0" (UID: "4347d253-53c6-4a1b-80a1-83a92f0202d0"). InnerVolumeSpecName "kube-api-access-dsmfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:18.569009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.568973    5958 generic.go:356] "Generic (PLEG): container finished" podID="5602cf8f-d9ef-412b-8901-91285525c030" containerID="e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3" exitCode=0
Oct 31 20:05:18.569201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.569036    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"
Oct 31 20:05:18.569201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.569061    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" event={"ID":"5602cf8f-d9ef-412b-8901-91285525c030","Type":"ContainerDied","Data":"e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3"}
Oct 31 20:05:18.569201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.569122    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj" event={"ID":"5602cf8f-d9ef-412b-8901-91285525c030","Type":"ContainerDied","Data":"4c9e7b2f4ceab7740f1acf25ac19dbf3ff5a9ea66380a62212b269605373e5c9"}
Oct 31 20:05:18.569201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.569157    5958 scope.go:122] "RemoveContainer" containerID="e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3"
Oct 31 20:05:18.571023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.570997    5958 generic.go:356] "Generic (PLEG): container finished" podID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerID="cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e" exitCode=0
Oct 31 20:05:18.571150 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.571039    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"
Oct 31 20:05:18.571150 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.571076    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" event={"ID":"4347d253-53c6-4a1b-80a1-83a92f0202d0","Type":"ContainerDied","Data":"cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e"}
Oct 31 20:05:18.571150 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.571122    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs" event={"ID":"4347d253-53c6-4a1b-80a1-83a92f0202d0","Type":"ContainerDied","Data":"a656abcc4f9a1496aed7f4b441276d62df81ec69c22f1552acf7bd412f4e88a9"}
Oct 31 20:05:18.584819 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.584745    5958 scope.go:122] "RemoveContainer" containerID="e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3"
Oct 31 20:05:18.585149 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:18.585122    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3\": container with ID starting with e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3 not found: ID does not exist" containerID="e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3"
Oct 31 20:05:18.585215 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.585161    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3"} err="failed to get container status \"e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3\": rpc error: code = NotFound desc = could not find container \"e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3\": container with ID starting with e72ad5e9503c9a6a93b14efb61158951ad497fcc343a542ec223b6fe80f767c3 not found: ID does not exist"
Oct 31 20:05:18.585215 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.585188    5958 scope.go:122] "RemoveContainer" containerID="cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e"
Oct 31 20:05:18.599800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.599723    5958 scope.go:122] "RemoveContainer" containerID="cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e"
Oct 31 20:05:18.601167 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:18.601115    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e\": container with ID starting with cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e not found: ID does not exist" containerID="cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e"
Oct 31 20:05:18.601167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.601154    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e"} err="failed to get container status \"cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e\": rpc error: code = NotFound desc = could not find container \"cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e\": container with ID starting with cb8dc9f23ee1cc697421a759a75573363f3908dfd306348495b410e95520c05e not found: ID does not exist"
Oct 31 20:05:18.611078 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.611018    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9c2n\" (UniqueName: \"kubernetes.io/projected/5602cf8f-d9ef-412b-8901-91285525c030-kube-api-access-k9c2n\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:18.611271 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.611084    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmfg\" (UniqueName: \"kubernetes.io/projected/4347d253-53c6-4a1b-80a1-83a92f0202d0-kube-api-access-dsmfg\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:18.623299 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.623266    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"]
Oct 31 20:05:18.627169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.627131    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-a891ce-5b8df746d7-qj9jj"]
Oct 31 20:05:18.643075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.642995    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"]
Oct 31 20:05:18.666870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.666835    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5602cf8f-d9ef-412b-8901-91285525c030" path="/var/lib/kubelet/pods/5602cf8f-d9ef-412b-8901-91285525c030/volumes"
Oct 31 20:05:18.667190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:18.667175    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-a891ce-f478d8c55-tz4rs"]
Oct 31 20:05:20.665158 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:20.665118    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" path="/var/lib/kubelet/pods/4347d253-53c6-4a1b-80a1-83a92f0202d0/volumes"
Oct 31 20:05:20.974556 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:20.974451    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:21.051836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.051797    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerName="payload-processing" containerID="cri-o://ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38" gracePeriod=25
Oct 31 20:05:21.071322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.071278    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerName="payload-pre-processing" containerID="cri-o://1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8" gracePeriod=25
Oct 31 20:05:21.419500 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.419464    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:21.437689 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.437658    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 20:05:21.555706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.555622    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.555706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.555671    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.555923 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.555792    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.555923 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.555826    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.556054 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.555913    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.556054 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.555962    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.556054 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556005    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.556054 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556047    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.556270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556128    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.556270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556153    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.556270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556215    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.556424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556404    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:21.556527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556506    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket\") pod \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\" (UID: \"e1fa028a-9dc6-4935-9456-1dc22bfb2578\") "
Oct 31 20:05:21.556869 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.556846    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.557315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.557290    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.557403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.557321    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.557403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.557337    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.557403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.557352    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.557403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.557365    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.558116 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.558080    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:21.558221 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.558145    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:21.558221 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.558184    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "kube-api-access-lwq7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:21.558349 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.558230    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy" pod "e1fa028a-9dc6-4935-9456-1dc22bfb2578" (UID: "e1fa028a-9dc6-4935-9456-1dc22bfb2578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.606866 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.606832    5958 generic.go:356] "Generic (PLEG): container finished" podID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerID="e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c" exitCode=0
Oct 31 20:05:21.607063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.606887    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"
Oct 31 20:05:21.607063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.606915    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" event={"ID":"e1fa028a-9dc6-4935-9456-1dc22bfb2578","Type":"ContainerDied","Data":"e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c"}
Oct 31 20:05:21.607063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.606972    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd" event={"ID":"e1fa028a-9dc6-4935-9456-1dc22bfb2578","Type":"ContainerDied","Data":"a1b7aaa06ecaed13b8c8398cdb5de5132ccf472fb6e0a10a530bca81bcfb79e4"}
Oct 31 20:05:21.607063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.606998    5958 scope.go:122] "RemoveContainer" containerID="e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c"
Oct 31 20:05:21.619746 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.619726    5958 scope.go:122] "RemoveContainer" containerID="e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"
Oct 31 20:05:21.630801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.630699    5958 scope.go:122] "RemoveContainer" containerID="e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c"
Oct 31 20:05:21.631262 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:21.631229    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c\": container with ID starting with e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c not found: ID does not exist" containerID="e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c"
Oct 31 20:05:21.631373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.631273    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c"} err="failed to get container status \"e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c\": rpc error: code = NotFound desc = could not find container \"e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c\": container with ID starting with e70387a6c4a2b644bfcd2551f50cff8336013239548ab7d2f12c0f58db3f749c not found: ID does not exist"
Oct 31 20:05:21.631373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.631300    5958 scope.go:122] "RemoveContainer" containerID="e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"
Oct 31 20:05:21.631668 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:21.631645    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1\": container with ID starting with e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1 not found: ID does not exist" containerID="e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"
Oct 31 20:05:21.631754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.631677    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1"} err="failed to get container status \"e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1\": rpc error: code = NotFound desc = could not find container \"e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1\": container with ID starting with e25868f2ee378ed11f7837658bfd0c509ca6368cac58558a1a693ef6d3fd4ed1 not found: ID does not exist"
Oct 31 20:05:21.638342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.638302    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"]
Oct 31 20:05:21.639306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.639282    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-gbjbd"]
Oct 31 20:05:21.658258 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.658227    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwq7p\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-kube-api-access-lwq7p\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.658258 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.658258    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.658460 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.658267    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.658460 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:21.658276    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fa028a-9dc6-4935-9456-1dc22bfb2578-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:22.426332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:22.426289    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:22.665495 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:22.665458    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" path="/var/lib/kubelet/pods/e1fa028a-9dc6-4935-9456-1dc22bfb2578/volumes"
Oct 31 20:05:22.716988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:22.716907    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:05:24.045578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.030926    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 20:05:24.130751 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.130711    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume\") pod \"d6fa35d0-5438-4dca-9ebe-ee270792971b\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") "
Oct 31 20:05:24.130968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.130956    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk\" (UniqueName: \"kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk\") pod \"d6fa35d0-5438-4dca-9ebe-ee270792971b\" (UID: \"d6fa35d0-5438-4dca-9ebe-ee270792971b\") "
Oct 31 20:05:24.131147 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.131121    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume" pod "d6fa35d0-5438-4dca-9ebe-ee270792971b" (UID: "d6fa35d0-5438-4dca-9ebe-ee270792971b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:24.131445 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.131426    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fa35d0-5438-4dca-9ebe-ee270792971b-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.133101 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.133080    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk" pod "d6fa35d0-5438-4dca-9ebe-ee270792971b" (UID: "d6fa35d0-5438-4dca-9ebe-ee270792971b"). InnerVolumeSpecName "kube-api-access-b5zpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:24.237378 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.237277    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zpk\" (UniqueName: \"kubernetes.io/projected/d6fa35d0-5438-4dca-9ebe-ee270792971b-kube-api-access-b5zpk\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.654991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.654958    5958 generic.go:356] "Generic (PLEG): container finished" podID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerID="b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561" exitCode=0
Oct 31 20:05:24.655189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.655004    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"
Oct 31 20:05:24.655189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.655037    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" event={"ID":"d6fa35d0-5438-4dca-9ebe-ee270792971b","Type":"ContainerDied","Data":"b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561"}
Oct 31 20:05:24.655189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.655090    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5" event={"ID":"d6fa35d0-5438-4dca-9ebe-ee270792971b","Type":"ContainerDied","Data":"e41f4b6444b00c63d582772de5b7406dc2a0b3049519a0da5eec3ab80df58373"}
Oct 31 20:05:24.655189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.655112    5958 scope.go:122] "RemoveContainer" containerID="b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561"
Oct 31 20:05:24.666398 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.666380    5958 scope.go:122] "RemoveContainer" containerID="b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561"
Oct 31 20:05:24.666732 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:24.666702    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561\": container with ID starting with b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561 not found: ID does not exist" containerID="b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561"
Oct 31 20:05:24.666856 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.666740    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561"} err="failed to get container status \"b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561\": rpc error: code = NotFound desc = could not find container \"b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561\": container with ID starting with b717ddf44d316b28e9f4d865f490f888ce205fc15475b4a1bcca87e3c0358561 not found: ID does not exist"
Oct 31 20:05:24.694471 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.694438    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"]
Oct 31 20:05:24.700673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:24.700644    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4bac5a-656d5db6dc-6mbl5"]
Oct 31 20:05:25.763093 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:25.763038    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main" containerID="cri-o://6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be" gracePeriod=45
Oct 31 20:05:26.051870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.051840    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 20:05:26.065503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065468    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache\") pod \"5a744336-2742-4cfe-8920-0119d2fffc85\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") "
Oct 31 20:05:26.065724 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065535    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj\" (UniqueName: \"kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj\") pod \"5a744336-2742-4cfe-8920-0119d2fffc85\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") "
Oct 31 20:05:26.065724 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065676    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm\") pod \"5a744336-2742-4cfe-8920-0119d2fffc85\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") "
Oct 31 20:05:26.065854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065771    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache" pod "5a744336-2742-4cfe-8920-0119d2fffc85" (UID: "5a744336-2742-4cfe-8920-0119d2fffc85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.065854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065799    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs\") pod \"5a744336-2742-4cfe-8920-0119d2fffc85\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") "
Oct 31 20:05:26.065854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065833    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir\") pod \"5a744336-2742-4cfe-8920-0119d2fffc85\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") "
Oct 31 20:05:26.066009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.065921    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home\") pod \"5a744336-2742-4cfe-8920-0119d2fffc85\" (UID: \"5a744336-2742-4cfe-8920-0119d2fffc85\") "
Oct 31 20:05:26.066156 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.066130    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir" pod "5a744336-2742-4cfe-8920-0119d2fffc85" (UID: "5a744336-2742-4cfe-8920-0119d2fffc85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.066247 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.066214    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home" pod "5a744336-2742-4cfe-8920-0119d2fffc85" (UID: "5a744336-2742-4cfe-8920-0119d2fffc85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.066483 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.066467    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.066527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.066485    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.066527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.066495    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.068412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.068377    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm" pod "5a744336-2742-4cfe-8920-0119d2fffc85" (UID: "5a744336-2742-4cfe-8920-0119d2fffc85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.068578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.068467    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs" pod "5a744336-2742-4cfe-8920-0119d2fffc85" (UID: "5a744336-2742-4cfe-8920-0119d2fffc85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:26.068578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.068496    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj" pod "5a744336-2742-4cfe-8920-0119d2fffc85" (UID: "5a744336-2742-4cfe-8920-0119d2fffc85"). InnerVolumeSpecName "kube-api-access-hjdlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:26.167557 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.167507    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a744336-2742-4cfe-8920-0119d2fffc85-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.167557 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.167536    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a744336-2742-4cfe-8920-0119d2fffc85-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.167823 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.167568    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdlj\" (UniqueName: \"kubernetes.io/projected/5a744336-2742-4cfe-8920-0119d2fffc85-kube-api-access-hjdlj\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.666248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.666219    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" path="/var/lib/kubelet/pods/d6fa35d0-5438-4dca-9ebe-ee270792971b/volumes"
Oct 31 20:05:26.672745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.672709    5958 generic.go:356] "Generic (PLEG): container finished" podID="5a744336-2742-4cfe-8920-0119d2fffc85" containerID="6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be" exitCode=0
Oct 31 20:05:26.672892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.672758    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"
Oct 31 20:05:26.672892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.672790    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" event={"ID":"5a744336-2742-4cfe-8920-0119d2fffc85","Type":"ContainerDied","Data":"6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be"}
Oct 31 20:05:26.672892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.672830    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk" event={"ID":"5a744336-2742-4cfe-8920-0119d2fffc85","Type":"ContainerDied","Data":"aea97301811f46a36a360f7e8921fa96b232f7486521b186c53c5a4c4bc2e5a3"}
Oct 31 20:05:26.672892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.672850    5958 scope.go:122] "RemoveContainer" containerID="6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be"
Oct 31 20:05:26.685082 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.685057    5958 scope.go:122] "RemoveContainer" containerID="6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be"
Oct 31 20:05:26.685448 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:26.685406    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be\": container with ID starting with 6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be not found: ID does not exist" containerID="6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be"
Oct 31 20:05:26.685581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.685447    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be"} err="failed to get container status \"6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be\": rpc error: code = NotFound desc = could not find container \"6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be\": container with ID starting with 6dead1c73d076341d1fcb5157361ce8cfdc1ee1288a5e1ecb5c97dcc40c896be not found: ID does not exist"
Oct 31 20:05:26.702348 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.702306    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"]
Oct 31 20:05:26.710181 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.710140    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-w4-996c32-kserve-7f848c74d5-4dfqk"]
Oct 31 20:05:26.858406 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.858372    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"]
Oct 31 20:05:26.873633 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:26.873602    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"]
Oct 31 20:05:28.666203 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:28.666166    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" path="/var/lib/kubelet/pods/5a744336-2742-4cfe-8920-0119d2fffc85/volumes"
Oct 31 20:05:28.943010 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:28.942907    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"]
Oct 31 20:05:28.943248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:28.943212    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx" podUID="61d3b347-992f-4d11-8358-1af3997f684e" containerName="maas-api" containerID="cri-o://89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205" gracePeriod=30
Oct 31 20:05:29.401615 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.401558    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 20:05:29.453182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.453143    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs\") pod \"61d3b347-992f-4d11-8358-1af3997f684e\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") "
Oct 31 20:05:29.453526 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.453495    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz\" (UniqueName: \"kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz\") pod \"61d3b347-992f-4d11-8358-1af3997f684e\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") "
Oct 31 20:05:29.453700 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.453634    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca\") pod \"61d3b347-992f-4d11-8358-1af3997f684e\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") "
Oct 31 20:05:29.453811 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.453741    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls\") pod \"61d3b347-992f-4d11-8358-1af3997f684e\" (UID: \"61d3b347-992f-4d11-8358-1af3997f684e\") "
Oct 31 20:05:29.454307 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.454283    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca" pod "61d3b347-992f-4d11-8358-1af3997f684e" (UID: "61d3b347-992f-4d11-8358-1af3997f684e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:29.460439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.460403    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls" pod "61d3b347-992f-4d11-8358-1af3997f684e" (UID: "61d3b347-992f-4d11-8358-1af3997f684e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:29.464025 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.462321    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs" pod "61d3b347-992f-4d11-8358-1af3997f684e" (UID: "61d3b347-992f-4d11-8358-1af3997f684e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:29.466320 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.466197    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz" pod "61d3b347-992f-4d11-8358-1af3997f684e" (UID: "61d3b347-992f-4d11-8358-1af3997f684e"). InnerVolumeSpecName "kube-api-access-xkgmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:29.555274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.555238    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgmz\" (UniqueName: \"kubernetes.io/projected/61d3b347-992f-4d11-8358-1af3997f684e-kube-api-access-xkgmz\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:29.555274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.555274    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d3b347-992f-4d11-8358-1af3997f684e-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:29.555274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.555290    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:29.555640 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.555306    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61d3b347-992f-4d11-8358-1af3997f684e-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:29.624599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.624529    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"]
Oct 31 20:05:29.699209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.698954    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"]
Oct 31 20:05:29.714630 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.714594    5958 generic.go:356] "Generic (PLEG): container finished" podID="61d3b347-992f-4d11-8358-1af3997f684e" containerID="89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205" exitCode=0
Oct 31 20:05:29.714968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.714666    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx" event={"ID":"61d3b347-992f-4d11-8358-1af3997f684e","Type":"ContainerDied","Data":"89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205"}
Oct 31 20:05:29.714968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.714705    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx" event={"ID":"61d3b347-992f-4d11-8358-1af3997f684e","Type":"ContainerDied","Data":"3d2ffe6d1fa8331ba14805f7e648453f9fce5eef6dc65c42d4ae8773e4f0d13b"}
Oct 31 20:05:29.714968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.714729    5958 scope.go:122] "RemoveContainer" containerID="89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205"
Oct 31 20:05:29.714968 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.714852    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"
Oct 31 20:05:29.741311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.741285    5958 scope.go:122] "RemoveContainer" containerID="89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205"
Oct 31 20:05:29.741724 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:29.741689    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205\": container with ID starting with 89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205 not found: ID does not exist" containerID="89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205"
Oct 31 20:05:29.741818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.741732    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205"} err="failed to get container status \"89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205\": rpc error: code = NotFound desc = could not find container \"89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205\": container with ID starting with 89a8af786f37e7d324b9315fb64460383029e8b1ecf512bf9dd4220e20615205 not found: ID does not exist"
Oct 31 20:05:29.769068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.769036    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"]
Oct 31 20:05:29.776827 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:29.776795    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e1d060-7cb848d98c-zprdx"]
Oct 31 20:05:30.548386 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.548352    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"]
Oct 31 20:05:30.548904 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.548624    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" podUID="0d76eb2a-ecc5-4b45-a134-754216e355c0" containerName="maas-api" containerID="cri-o://7b458b6d03cc70bd6bc6265034af816c7936f526699c0a5cad774c14f1a0b74f" gracePeriod=30
Oct 31 20:05:30.667183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.666987    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d3b347-992f-4d11-8358-1af3997f684e" path="/var/lib/kubelet/pods/61d3b347-992f-4d11-8358-1af3997f684e/volumes"
Oct 31 20:05:30.724215 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.724186    5958 generic.go:356] "Generic (PLEG): container finished" podID="0d76eb2a-ecc5-4b45-a134-754216e355c0" containerID="7b458b6d03cc70bd6bc6265034af816c7936f526699c0a5cad774c14f1a0b74f" exitCode=0
Oct 31 20:05:30.724654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.724259    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" event={"ID":"0d76eb2a-ecc5-4b45-a134-754216e355c0","Type":"ContainerDied","Data":"7b458b6d03cc70bd6bc6265034af816c7936f526699c0a5cad774c14f1a0b74f"}
Oct 31 20:05:30.921628 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.921539    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 20:05:30.997902 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.997863    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"]
Oct 31 20:05:30.999199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999167    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main"
Oct 31 20:05:30.999199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999197    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999224    5958 cpu_manager.go:414] "containerMap: removing container" podUID="607f4e4f-8aad-42d8-a162-f4235675ccac" containerName="maas-api"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999233    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607f4e4f-8aad-42d8-a162-f4235675ccac" containerName="maas-api"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999249    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" containerName="payload-pre-processing"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999258    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" containerName="payload-pre-processing"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999277    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999286    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999300    5958 cpu_manager.go:414] "containerMap: removing container" podUID="61d3b347-992f-4d11-8358-1af3997f684e" containerName="maas-api"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999308    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d3b347-992f-4d11-8358-1af3997f684e" containerName="maas-api"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999322    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0d76eb2a-ecc5-4b45-a134-754216e355c0" containerName="maas-api"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999330    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d76eb2a-ecc5-4b45-a134-754216e355c0" containerName="maas-api"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999340    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerName="payload-pre-processing"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999349    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerName="payload-pre-processing"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999360    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999368    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy"
Oct 31 20:05:30.999374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999377    5958 cpu_manager.go:414] "containerMap: removing container" podUID="37467c1d-a0f5-462d-a9ee-4e98ef27713f" containerName="payload-pre-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999383    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37467c1d-a0f5-462d-a9ee-4e98ef27713f" containerName="payload-pre-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999396    5958 cpu_manager.go:414] "containerMap: removing container" podUID="96842103-1def-4b3a-9321-2e4434e33a5b" containerName="payload-pre-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999404    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96842103-1def-4b3a-9321-2e4434e33a5b" containerName="payload-pre-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999414    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5602cf8f-d9ef-412b-8901-91285525c030" containerName="payload-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999423    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5602cf8f-d9ef-412b-8901-91285525c030" containerName="payload-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999445    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999451    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999468    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999474    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999502    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerName="payload-pre-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999508    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerName="payload-pre-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999538    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f0f562b3-c37e-4710-a571-614799454fc6" containerName="payload-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999562    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f562b3-c37e-4710-a571-614799454fc6" containerName="payload-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999595    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999601    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999633    5958 cpu_manager.go:414] "containerMap: removing container" podUID="40045daf-6b06-40e0-90ee-aac67220d7b7" containerName="maas-api"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999639    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40045daf-6b06-40e0-90ee-aac67220d7b7" containerName="maas-api"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999650    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" containerName="payload-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999658    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" containerName="payload-processing"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999674    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c04b6288-9859-4958-9bc1-4104c779a292" containerName="maas-api"
Oct 31 20:05:30.999850 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999681    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04b6288-9859-4958-9bc1-4104c779a292" containerName="maas-api"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999890    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f562b3-c37e-4710-a571-614799454fc6" containerName="payload-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999910    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5eec6e7-f7ed-4db8-ae0e-7a3e6958b1d3" containerName="istio-proxy"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999927    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="40045daf-6b06-40e0-90ee-aac67220d7b7" containerName="maas-api"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999937    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d461c42-cd97-42d7-9098-b4911126ce01" containerName="payload-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999948    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04b6288-9859-4958-9bc1-4104c779a292" containerName="maas-api"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999962    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a744336-2742-4cfe-8920-0119d2fffc85" containerName="main"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999974    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="607f4e4f-8aad-42d8-a162-f4235675ccac" containerName="maas-api"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999984    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66eba36-4f37-41dd-86e0-35987e97506d" containerName="payload-pre-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:30.999997    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9193f73f-a5b0-4033-ac97-94c990e4019c" containerName="istio-proxy"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000006    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d3b347-992f-4d11-8358-1af3997f684e" containerName="maas-api"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000016    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d76eb2a-ecc5-4b45-a134-754216e355c0" containerName="maas-api"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000025    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4347d253-53c6-4a1b-80a1-83a92f0202d0" containerName="payload-pre-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000037    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000049    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="37467c1d-a0f5-462d-a9ee-4e98ef27713f" containerName="payload-pre-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000058    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="96842103-1def-4b3a-9321-2e4434e33a5b" containerName="payload-pre-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000068    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5602cf8f-d9ef-412b-8901-91285525c030" containerName="payload-processing"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000080    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9284c37-4d8c-4e02-985f-b63923457bc2" containerName="istio-proxy"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000090    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1fa028a-9dc6-4935-9456-1dc22bfb2578" containerName="istio-proxy"
Oct 31 20:05:31.000473 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.000101    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fa35d0-5438-4dca-9ebe-ee270792971b" containerName="payload-pre-processing"
Oct 31 20:05:31.006981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.006900    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.035744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.035708    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"]
Oct 31 20:05:31.092439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.092354    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs\") pod \"0d76eb2a-ecc5-4b45-a134-754216e355c0\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") "
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.092737    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h\" (UniqueName: \"kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h\") pod \"0d76eb2a-ecc5-4b45-a134-754216e355c0\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") "
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.092802    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca\") pod \"0d76eb2a-ecc5-4b45-a134-754216e355c0\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") "
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094161    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\") pod \"0d76eb2a-ecc5-4b45-a134-754216e355c0\" (UID: \"0d76eb2a-ecc5-4b45-a134-754216e355c0\") "
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094601    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvk7k\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094659    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094770    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094823    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.094985    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.095003    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.095036    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.095124    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.095589    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca" pod "0d76eb2a-ecc5-4b45-a134-754216e355c0" (UID: "0d76eb2a-ecc5-4b45-a134-754216e355c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:31.099056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.098997    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f28d48-openshift-default-dockercfg-mg2rh\""
Oct 31 20:05:31.100214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.100158    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls" pod "0d76eb2a-ecc5-4b45-a134-754216e355c0" (UID: "0d76eb2a-ecc5-4b45-a134-754216e355c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:31.100214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.100177    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs" pod "0d76eb2a-ecc5-4b45-a134-754216e355c0" (UID: "0d76eb2a-ecc5-4b45-a134-754216e355c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:31.100918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.100890    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h" pod "0d76eb2a-ecc5-4b45-a134-754216e355c0" (UID: "0d76eb2a-ecc5-4b45-a134-754216e355c0"). InnerVolumeSpecName "kube-api-access-h7v7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:31.196907 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.196872    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.196918    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.196961    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197013    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197035    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197064    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197117    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197210    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvk7k\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197240    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197299    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:31.197449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197310    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7v7h\" (UniqueName: \"kubernetes.io/projected/0d76eb2a-ecc5-4b45-a134-754216e355c0-kube-api-access-h7v7h\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:31.197449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197321    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d76eb2a-ecc5-4b45-a134-754216e355c0-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:31.197449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197335    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d76eb2a-ecc5-4b45-a134-754216e355c0-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:31.197769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197664    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197697    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.197959 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.197888    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.198593 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.198199    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.200280 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.199263    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.201494 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.201472    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.201714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.201693    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.216451 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.216419    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvk7k\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.223978 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.223953    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.328793 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.328766    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:31.419946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.419906    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:31.638832 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.638796    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"]
Oct 31 20:05:31.639021 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:05:31.638884    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2140f87_1dde_4114_924f_7eb27f801109.slice/crio-1569f8051d202134b89b7cfd7294828e6323a5be3a2ff6e474689524708eac53 WatchSource:0}: Error finding container 1569f8051d202134b89b7cfd7294828e6323a5be3a2ff6e474689524708eac53: Status 404 returned error can't find the container with id 1569f8051d202134b89b7cfd7294828e6323a5be3a2ff6e474689524708eac53
Oct 31 20:05:31.642056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.642019    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:05:31.642149 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.642110    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:05:31.642186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.642156    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:05:31.735097 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.735067    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"
Oct 31 20:05:31.735413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.735104    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn" event={"ID":"0d76eb2a-ecc5-4b45-a134-754216e355c0","Type":"ContainerDied","Data":"24f34050d935ad87ba3bdf1b12d018f106bfdce8c5b6ce27902cfc3bd270d580"}
Oct 31 20:05:31.735413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.735175    5958 scope.go:122] "RemoveContainer" containerID="7b458b6d03cc70bd6bc6265034af816c7936f526699c0a5cad774c14f1a0b74f"
Oct 31 20:05:31.741292 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.741258    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" event={"ID":"a2140f87-1dde-4114-924f-7eb27f801109","Type":"ContainerStarted","Data":"1569f8051d202134b89b7cfd7294828e6323a5be3a2ff6e474689524708eac53"}
Oct 31 20:05:31.788578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.788527    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"]
Oct 31 20:05:31.803824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.803793    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ae2e60-6b5475dd58-bhchn"]
Oct 31 20:05:31.859122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.859074    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" podUID="b326360d-426f-4051-8d04-7856458d4950" containerName="payload-processing" containerID="cri-o://b1426ee79149ff92c8e03e69e5daf7470727ebe3eb1e0301de0f0daaac8cda43" gracePeriod=25
Oct 31 20:05:31.874058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:31.874016    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerName="payload-pre-processing" containerID="cri-o://f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2" gracePeriod=25
Oct 31 20:05:32.427142 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:32.427105    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:32.665534 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:32.665503    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d76eb2a-ecc5-4b45-a134-754216e355c0" path="/var/lib/kubelet/pods/0d76eb2a-ecc5-4b45-a134-754216e355c0/volumes"
Oct 31 20:05:32.750899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:32.750805    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" event={"ID":"a2140f87-1dde-4114-924f-7eb27f801109","Type":"ContainerStarted","Data":"4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71"}
Oct 31 20:05:32.781342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:32.781292    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podStartSLOduration=2.781275981 podStartE2EDuration="2.781275981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:32.780494489 +0000 UTC m=+2726.241869106" watchObservedRunningTime="2026-10-31 20:05:32.781275981 +0000 UTC m=+2726.242650641"
Oct 31 20:05:33.330047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.330010    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:33.332393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.332363    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Oct 31 20:05:33.332528 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.332416    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Oct 31 20:05:33.462214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.462184    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"]
Oct 31 20:05:33.466899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.466874    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.480102 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.480058    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7fe845-dockercfg-hbttf\""
Oct 31 20:05:33.481939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.481913    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7fe845\""
Oct 31 20:05:33.485057 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.485032    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"]
Oct 31 20:05:33.537238 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.537205    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"]
Oct 31 20:05:33.542134 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.542108    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.546846 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.546602    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.549677 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.549644    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.549983 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.549965    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpw49\" (UniqueName: \"kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.550184 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.550168    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c56xr\" (UniqueName: \"kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr\") pod \"payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.561566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.555018    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"]
Oct 31 20:05:33.678566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.677522    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.678566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.677664    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpw49\" (UniqueName: \"kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.678566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.677694    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c56xr\" (UniqueName: \"kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr\") pod \"payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.678566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.677813    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.697507 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.697471    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.697713 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.697561    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.781012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.780985    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpw49\" (UniqueName: \"kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.781438 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.781398    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c56xr\" (UniqueName: \"kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr\") pod \"payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:33.783531 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.783507    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:33.875269 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:33.875235    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:34.101503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.101467    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"]
Oct 31 20:05:34.250989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.250960    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"]
Oct 31 20:05:34.252124 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:05:34.252097    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84222e00_b33d_4828_839e_b4eefe6c5c64.slice/crio-f36893793fd0ccf6697d8b83e7b7814a7eba403f01913e2e48251fccab2a5ab4 WatchSource:0}: Error finding container f36893793fd0ccf6697d8b83e7b7814a7eba403f01913e2e48251fccab2a5ab4: Status 404 returned error can't find the container with id f36893793fd0ccf6697d8b83e7b7814a7eba403f01913e2e48251fccab2a5ab4
Oct 31 20:05:34.330402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.329941    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Oct 31 20:05:34.330402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.330002    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Oct 31 20:05:34.628663 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.628614    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerName="payload-processing" containerID="cri-o://002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325" gracePeriod=25
Oct 31 20:05:34.700169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.700124    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerName="payload-pre-processing" containerID="cri-o://9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a" gracePeriod=25
Oct 31 20:05:34.788859 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.788818    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm" event={"ID":"3af08b8b-8eea-409a-8573-3468870673c9","Type":"ContainerStarted","Data":"3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14"}
Oct 31 20:05:34.789334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.788891    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm" event={"ID":"3af08b8b-8eea-409a-8573-3468870673c9","Type":"ContainerStarted","Data":"a885592a9ec58588e74b3d0fcd92d4f80075d2b11470585a86ff7e7debc879f2"}
Oct 31 20:05:34.789334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.789019    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:34.790515 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.790492    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn" event={"ID":"84222e00-b33d-4828-839e-b4eefe6c5c64","Type":"ContainerStarted","Data":"4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13"}
Oct 31 20:05:34.790669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.790586    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:34.790669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.790630    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn" event={"ID":"84222e00-b33d-4828-839e-b4eefe6c5c64","Type":"ContainerStarted","Data":"f36893793fd0ccf6697d8b83e7b7814a7eba403f01913e2e48251fccab2a5ab4"}
Oct 31 20:05:34.833587 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.833479    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm" podStartSLOduration=1.833459572 podStartE2EDuration="1.833459572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:34.823840842 +0000 UTC m=+2728.285215458" watchObservedRunningTime="2026-10-31 20:05:34.833459572 +0000 UTC m=+2728.294834189"
Oct 31 20:05:34.850408 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:34.850356    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn" podStartSLOduration=1.850342494 podStartE2EDuration="1.850342494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:34.849079589 +0000 UTC m=+2728.310454204" watchObservedRunningTime="2026-10-31 20:05:34.850342494 +0000 UTC m=+2728.311717130"
Oct 31 20:05:35.329949 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:35.329912    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Oct 31 20:05:35.330244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:35.330211    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Oct 31 20:05:36.330268 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:36.330222    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Oct 31 20:05:36.330807 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:36.330301    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Oct 31 20:05:37.331570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:37.330712    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body=
Oct 31 20:05:37.331570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:37.330785    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused"
Oct 31 20:05:38.389996 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:38.389962    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:38.390519 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:38.390495    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:38.391607 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:38.391563    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:05:39.133131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:39.133098    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"]
Oct 31 20:05:39.133379 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:39.133352    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" containerID="cri-o://6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478" gracePeriod=30
Oct 31 20:05:39.504317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:39.504209    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:39.504317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:39.504210    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" podUID="b326360d-426f-4051-8d04-7856458d4950" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:40.921146 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:40.921100    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:40.921612 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:40.921146    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:41.071231 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.068569    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"]
Oct 31 20:05:41.071231 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.068916    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" containerID="cri-o://ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60" gracePeriod=30
Oct 31 20:05:41.076211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.076182    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"]
Oct 31 20:05:41.076474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.076443    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" containerID="cri-o://09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e" gracePeriod=30
Oct 31 20:05:41.088319 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.088286    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"]
Oct 31 20:05:41.088612 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.088537    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" containerID="cri-o://6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d" gracePeriod=30
Oct 31 20:05:41.411341 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.411315    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 20:05:41.447379 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.447340    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx\" (UniqueName: \"kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx\") pod \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") "
Oct 31 20:05:41.447595 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.447492    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume\") pod \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\" (UID: \"1fad730e-f46a-4e09-93e7-3bbf2b46aea6\") "
Oct 31 20:05:41.448175 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.448148    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume" pod "1fad730e-f46a-4e09-93e7-3bbf2b46aea6" (UID: "1fad730e-f46a-4e09-93e7-3bbf2b46aea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:41.449963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.449938    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx" pod "1fad730e-f46a-4e09-93e7-3bbf2b46aea6" (UID: "1fad730e-f46a-4e09-93e7-3bbf2b46aea6"). InnerVolumeSpecName "kube-api-access-hk4fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:41.481503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.481476    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 20:05:41.548525 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.548491    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume\") pod \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") "
Oct 31 20:05:41.548751 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.548615    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9\" (UniqueName: \"kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9\") pod \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\" (UID: \"dee1c945-8aaa-4d51-849f-dac9021fcbc1\") "
Oct 31 20:05:41.549130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.549110    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk4fx\" (UniqueName: \"kubernetes.io/projected/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-kube-api-access-hk4fx\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:41.549222 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.549131    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fad730e-f46a-4e09-93e7-3bbf2b46aea6-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:41.549284 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.549231    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume" pod "dee1c945-8aaa-4d51-849f-dac9021fcbc1" (UID: "dee1c945-8aaa-4d51-849f-dac9021fcbc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:41.551109 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.551082    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9" pod "dee1c945-8aaa-4d51-849f-dac9021fcbc1" (UID: "dee1c945-8aaa-4d51-849f-dac9021fcbc1"). InnerVolumeSpecName "kube-api-access-pvrp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:41.650629 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.650590    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrp9\" (UniqueName: \"kubernetes.io/projected/dee1c945-8aaa-4d51-849f-dac9021fcbc1-kube-api-access-pvrp9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:41.650629 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.650624    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee1c945-8aaa-4d51-849f-dac9021fcbc1-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:41.865864 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.865768    5958 generic.go:356] "Generic (PLEG): container finished" podID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerID="ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38" exitCode=0
Oct 31 20:05:41.865864 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.865820    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"
Oct 31 20:05:41.865864 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.865847    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" event={"ID":"1fad730e-f46a-4e09-93e7-3bbf2b46aea6","Type":"ContainerDied","Data":"ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38"}
Oct 31 20:05:41.866150 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.865900    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv" event={"ID":"1fad730e-f46a-4e09-93e7-3bbf2b46aea6","Type":"ContainerDied","Data":"3776670b1194d49df9862bd0f34b9ed6ddf55d684d70a4a26febb8eb4ba7062d"}
Oct 31 20:05:41.866150 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.865923    5958 scope.go:122] "RemoveContainer" containerID="ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38"
Oct 31 20:05:41.867668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.867640    5958 generic.go:356] "Generic (PLEG): container finished" podID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerID="1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8" exitCode=0
Oct 31 20:05:41.867801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.867691    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"
Oct 31 20:05:41.867801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.867716    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" event={"ID":"dee1c945-8aaa-4d51-849f-dac9021fcbc1","Type":"ContainerDied","Data":"1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8"}
Oct 31 20:05:41.867801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.867765    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt" event={"ID":"dee1c945-8aaa-4d51-849f-dac9021fcbc1","Type":"ContainerDied","Data":"2bf27f6d7b0761696064509ac29f37b82406418710645800d576d7e317da7571"}
Oct 31 20:05:41.877997 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.877874    5958 scope.go:122] "RemoveContainer" containerID="ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38"
Oct 31 20:05:41.879810 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:41.879786    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38\": container with ID starting with ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38 not found: ID does not exist" containerID="ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38"
Oct 31 20:05:41.879923 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.879816    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38"} err="failed to get container status \"ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38\": rpc error: code = NotFound desc = could not find container \"ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38\": container with ID starting with ae6d3d1de55db1c286e36803310e85993ebd8c83b5d712387240fed5181f5d38 not found: ID does not exist"
Oct 31 20:05:41.879923 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.879833    5958 scope.go:122] "RemoveContainer" containerID="1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8"
Oct 31 20:05:41.889964 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.889939    5958 scope.go:122] "RemoveContainer" containerID="1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8"
Oct 31 20:05:41.890243 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:41.890217    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8\": container with ID starting with 1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8 not found: ID does not exist" containerID="1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8"
Oct 31 20:05:41.890339 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.890252    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8"} err="failed to get container status \"1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8\": rpc error: code = NotFound desc = could not find container \"1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8\": container with ID starting with 1182b3a051506487ae32159842180a873b6d8163129ea3c410a635b199db93a8 not found: ID does not exist"
Oct 31 20:05:41.897829 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.897803    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"]
Oct 31 20:05:41.912867 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.912835    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-996c32-7585565cff-4bwmt"]
Oct 31 20:05:41.928068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.928037    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"]
Oct 31 20:05:41.940483 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:41.940455    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-996c32-5b66846755-x5jkv"]
Oct 31 20:05:42.428614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.428502    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:42.428614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.428580    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:42.520635 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.520599    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:42.520799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.520658    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:42.640357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.640319    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:42.640522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.640393    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:42.664749 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.664716    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" path="/var/lib/kubelet/pods/1fad730e-f46a-4e09-93e7-3bbf2b46aea6/volumes"
Oct 31 20:05:42.665199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.665184    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" path="/var/lib/kubelet/pods/dee1c945-8aaa-4d51-849f-dac9021fcbc1/volumes"
Oct 31 20:05:42.983094 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.983060    5958 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:42.983470 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:42.983118    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:43.733916 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.733869    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"]
Oct 31 20:05:43.734844 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.734821    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerName="payload-processing"
Oct 31 20:05:43.734844 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.734845    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerName="payload-processing"
Oct 31 20:05:43.735050 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.734880    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerName="payload-pre-processing"
Oct 31 20:05:43.735050 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.734888    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerName="payload-pre-processing"
Oct 31 20:05:43.735164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.735076    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fad730e-f46a-4e09-93e7-3bbf2b46aea6" containerName="payload-processing"
Oct 31 20:05:43.735164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.735095    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee1c945-8aaa-4d51-849f-dac9021fcbc1" containerName="payload-pre-processing"
Oct 31 20:05:43.740067 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.740043    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.759762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.759733    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"]
Oct 31 20:05:43.772747 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.772717    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7fe845-serving-cert\""
Oct 31 20:05:43.784988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.784950    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.785218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.785192    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.785297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.785266    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4flb5\" (UniqueName: \"kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.785370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.785354    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.886912 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.886873    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.887122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.886997    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.887122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.887041    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4flb5\" (UniqueName: \"kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.887122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.887108    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.887872 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.887844    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.889737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.889710    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.889855 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.889710    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:43.905297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:43.900840    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4flb5\" (UniqueName: \"kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5\") pod \"maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:44.074239 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.074157    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:44.212231 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:44.212191    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847c2db_ec3f_4e41_8140_5e7a9db18309.slice/crio-6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:05:44.410165 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:05:44.410128    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5730e849_2fe7_41a5_894f_3def0ee02cb8.slice/crio-c34f1b1e1c24006e2bbd762b4baf68e0044bdb917db79a939fc563c4cf4bdb66 WatchSource:0}: Error finding container c34f1b1e1c24006e2bbd762b4baf68e0044bdb917db79a939fc563c4cf4bdb66: Status 404 returned error can't find the container with id c34f1b1e1c24006e2bbd762b4baf68e0044bdb917db79a939fc563c4cf4bdb66
Oct 31 20:05:44.413881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.412960    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"]
Oct 31 20:05:44.428696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.428665    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.495677    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.495752    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.495803    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.495862    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.495937    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.495987    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.496018    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.496082    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.496130    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data\") pod \"a847c2db-ec3f-4e41-8140-5e7a9db18309\" (UID: \"a847c2db-ec3f-4e41-8140-5e7a9db18309\") "
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.498924    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.499143    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.499510    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.500159    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.502572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.500748    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.513729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.504267    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:44.513729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.505958    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "kube-api-access-t579d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:44.513729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.510348    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.513729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.513680    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo" pod "a847c2db-ec3f-4e41-8140-5e7a9db18309" (UID: "a847c2db-ec3f-4e41-8140-5e7a9db18309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:44.600098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600058    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600090    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847c2db-ec3f-4e41-8140-5e7a9db18309-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600100    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600111    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600120    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600130    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600138    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600147    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t579d\" (UniqueName: \"kubernetes.io/projected/a847c2db-ec3f-4e41-8140-5e7a9db18309-kube-api-access-t579d\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.600345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.600157    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847c2db-ec3f-4e41-8140-5e7a9db18309-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.639323 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.639288    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"]
Oct 31 20:05:44.647427 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.647345    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"]
Oct 31 20:05:44.907526 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.907445    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" event={"ID":"5730e849-2fe7-41a5-894f-3def0ee02cb8","Type":"ContainerStarted","Data":"c34f1b1e1c24006e2bbd762b4baf68e0044bdb917db79a939fc563c4cf4bdb66"}
Oct 31 20:05:44.909162 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.909133    5958 generic.go:356] "Generic (PLEG): container finished" podID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerID="6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478" exitCode=0
Oct 31 20:05:44.909306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.909182    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"
Oct 31 20:05:44.909306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.909214    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" event={"ID":"a847c2db-ec3f-4e41-8140-5e7a9db18309","Type":"ContainerDied","Data":"6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478"}
Oct 31 20:05:44.909306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.909259    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng" event={"ID":"a847c2db-ec3f-4e41-8140-5e7a9db18309","Type":"ContainerDied","Data":"1edce90f748c835bc05850076691e7cacd6ceb56eac08c78259f0e7e04cce303"}
Oct 31 20:05:44.909306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.909282    5958 scope.go:122] "RemoveContainer" containerID="6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478"
Oct 31 20:05:44.929262 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.929241    5958 scope.go:122] "RemoveContainer" containerID="6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478"
Oct 31 20:05:44.929602 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:44.929578    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478\": container with ID starting with 6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478 not found: ID does not exist" containerID="6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478"
Oct 31 20:05:44.929694 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.929611    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478"} err="failed to get container status \"6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478\": rpc error: code = NotFound desc = could not find container \"6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478\": container with ID starting with 6e030f6bc6a37687dc8f0b11d53ffd73029e4641b3f83234294c289248a28478 not found: ID does not exist"
Oct 31 20:05:44.958250 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.958191    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"]
Oct 31 20:05:44.961465 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:44.961441    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-6zgng"]
Oct 31 20:05:45.803702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:45.803675    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:05:45.804190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:45.803759    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:05:45.921504 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:45.921458    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" event={"ID":"5730e849-2fe7-41a5-894f-3def0ee02cb8","Type":"ContainerStarted","Data":"98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571"}
Oct 31 20:05:45.921708 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:45.921568    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:45.953117 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:45.953054    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" podStartSLOduration=2.503391707 podStartE2EDuration="2.95303673s" totalImagesPullingTime="449.645023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:45.951488456 +0000 UTC m=+2739.412863072" watchObservedRunningTime="2026-10-31 20:05:45.95303673 +0000 UTC m=+2739.414411346"
Oct 31 20:05:46.302821 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.302789    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 20:05:46.324647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.324608    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.324841 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.324714    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.324841 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.324754    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.324841 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.324834    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.325012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.324916    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.325012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.324972    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.325012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.325001    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.325173 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.325076    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.325173 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.325155    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket\") pod \"83a1856b-cd18-4bfb-9a25-68175c3858e6\" (UID: \"83a1856b-cd18-4bfb-9a25-68175c3858e6\") "
Oct 31 20:05:46.325317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.325289    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.325676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.325649    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.325810 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.325792    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.326697 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.326657    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.327065 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.327042    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.327389 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.327370    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.346074 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.345735    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:46.346074 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.345982    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.346507 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.346468    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "kube-api-access-4gn8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.351213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.351172    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy" pod "83a1856b-cd18-4bfb-9a25-68175c3858e6" (UID: "83a1856b-cd18-4bfb-9a25-68175c3858e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.376927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.376899    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 20:05:46.427005 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.426970    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427210 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427055    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427210 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427172    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427210 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427203    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427437 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427243    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427437 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427279    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427437 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427311    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.427711 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.427688    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 20:05:46.428379 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.428112    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.428379 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.428181    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.428379 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.428312    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.428668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.428454    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.428668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.428563    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs\") pod \"cdec16c6-aa60-40ff-a953-e3c291e05e24\" (UID: \"cdec16c6-aa60-40ff-a953-e3c291e05e24\") "
Oct 31 20:05:46.429322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429211    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.429322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429258    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429276    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429290    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429303    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429316    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a1856b-cd18-4bfb-9a25-68175c3858e6-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429346    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gn8p\" (UniqueName: \"kubernetes.io/projected/83a1856b-cd18-4bfb-9a25-68175c3858e6-kube-api-access-4gn8p\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429361    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429376    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdec16c6-aa60-40ff-a953-e3c291e05e24-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429389    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429401    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a1856b-cd18-4bfb-9a25-68175c3858e6-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.429744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.429413    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a1856b-cd18-4bfb-9a25-68175c3858e6-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.430266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.430194    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.431331 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.431305    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "kube-api-access-j9clf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.433158 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.433135    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.437186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.437152    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:46.438373 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.438352    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy" pod "cdec16c6-aa60-40ff-a953-e3c291e05e24" (UID: "cdec16c6-aa60-40ff-a953-e3c291e05e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.530271 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530221    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530301    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530348    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530393    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530419    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530508    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530627    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.530769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530732    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.531086 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.530787    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data\") pod \"0592a59d-4a48-432c-acef-7dab11bf3809\" (UID: \"0592a59d-4a48-432c-acef-7dab11bf3809\") "
Oct 31 20:05:46.531396 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531368    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.531396 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531393    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.531626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531408    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.531626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531422    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9clf\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-kube-api-access-j9clf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.531626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531436    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdec16c6-aa60-40ff-a953-e3c291e05e24-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.531626 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531450    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdec16c6-aa60-40ff-a953-e3c291e05e24-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.531929 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.531727    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.532085 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.532049    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.532296 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.532277    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.533359 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.533327    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.534166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.534147    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.536202 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.536161    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:46.538820 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.538784    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.538949 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.538877    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.539023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.538946    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55" pod "0592a59d-4a48-432c-acef-7dab11bf3809" (UID: "0592a59d-4a48-432c-acef-7dab11bf3809"). InnerVolumeSpecName "kube-api-access-zns55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.540630 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.540563    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"]
Oct 31 20:05:46.633325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633292    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zns55\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-kube-api-access-zns55\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633322    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0592a59d-4a48-432c-acef-7dab11bf3809-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633325 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633335    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0592a59d-4a48-432c-acef-7dab11bf3809-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633344    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633353    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633362    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633370    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633378    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0592a59d-4a48-432c-acef-7dab11bf3809-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.633608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.633388    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0592a59d-4a48-432c-acef-7dab11bf3809-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.667137 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.667102    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" path="/var/lib/kubelet/pods/a847c2db-ec3f-4e41-8140-5e7a9db18309/volumes"
Oct 31 20:05:46.930388 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.930286    5958 generic.go:356] "Generic (PLEG): container finished" podID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerID="09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e" exitCode=0
Oct 31 20:05:46.930388 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.930347    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"
Oct 31 20:05:46.930388 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.930367    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" event={"ID":"cdec16c6-aa60-40ff-a953-e3c291e05e24","Type":"ContainerDied","Data":"09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e"}
Oct 31 20:05:46.931080 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.930422    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8" event={"ID":"cdec16c6-aa60-40ff-a953-e3c291e05e24","Type":"ContainerDied","Data":"91074c33675e770cab13f66305db4b3f242816f971c1e61db41e13d1e94fad21"}
Oct 31 20:05:46.931080 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.930445    5958 scope.go:122] "RemoveContainer" containerID="09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e"
Oct 31 20:05:46.932327 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.932302    5958 generic.go:356] "Generic (PLEG): container finished" podID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerID="ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60" exitCode=0
Oct 31 20:05:46.932459 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.932350    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"
Oct 31 20:05:46.932459 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.932387    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" event={"ID":"83a1856b-cd18-4bfb-9a25-68175c3858e6","Type":"ContainerDied","Data":"ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60"}
Oct 31 20:05:46.932459 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.932432    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b" event={"ID":"83a1856b-cd18-4bfb-9a25-68175c3858e6","Type":"ContainerDied","Data":"bcbb279d950f731f6de4b612a4c2423d932ee025eaa3e721da7dded850cdadc8"}
Oct 31 20:05:46.934365 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.934340    5958 generic.go:356] "Generic (PLEG): container finished" podID="0592a59d-4a48-432c-acef-7dab11bf3809" containerID="6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d" exitCode=0
Oct 31 20:05:46.934478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.934385    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"
Oct 31 20:05:46.934478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.934421    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" event={"ID":"0592a59d-4a48-432c-acef-7dab11bf3809","Type":"ContainerDied","Data":"6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d"}
Oct 31 20:05:46.934478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.934469    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds" event={"ID":"0592a59d-4a48-432c-acef-7dab11bf3809","Type":"ContainerDied","Data":"6abbe792c6a11b753697270d7b084121854094fc2d98a71a898339baef3564ff"}
Oct 31 20:05:46.943810 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.943792    5958 scope.go:122] "RemoveContainer" containerID="09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e"
Oct 31 20:05:46.944126 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:46.944104    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e\": container with ID starting with 09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e not found: ID does not exist" containerID="09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e"
Oct 31 20:05:46.944193 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.944135    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e"} err="failed to get container status \"09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e\": rpc error: code = NotFound desc = could not find container \"09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e\": container with ID starting with 09f16cd3db9a8bf25071c7bffddfd5e62be2b90e9e638464ea3ef86fab53ff9e not found: ID does not exist"
Oct 31 20:05:46.944193 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.944151    5958 scope.go:122] "RemoveContainer" containerID="ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60"
Oct 31 20:05:46.954650 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.954631    5958 scope.go:122] "RemoveContainer" containerID="ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60"
Oct 31 20:05:46.954915 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:46.954892    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60\": container with ID starting with ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60 not found: ID does not exist" containerID="ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60"
Oct 31 20:05:46.954970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.954922    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60"} err="failed to get container status \"ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60\": rpc error: code = NotFound desc = could not find container \"ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60\": container with ID starting with ce05c4ef59f00f504e766196bc12fa9d7729d3d4126b9ab7f9c896fc4a41aa60 not found: ID does not exist"
Oct 31 20:05:46.954970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.954942    5958 scope.go:122] "RemoveContainer" containerID="6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d"
Oct 31 20:05:46.965952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.965929    5958 scope.go:122] "RemoveContainer" containerID="6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d"
Oct 31 20:05:46.966260 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:46.966239    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d\": container with ID starting with 6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d not found: ID does not exist" containerID="6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d"
Oct 31 20:05:46.966352 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.966271    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d"} err="failed to get container status \"6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d\": rpc error: code = NotFound desc = could not find container \"6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d\": container with ID starting with 6a5df91fb5a2b01dd8ea782aa48f77873f8fafb48a8e6e51b23812a1fdea9d4d not found: ID does not exist"
Oct 31 20:05:46.968021 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.968001    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"]
Oct 31 20:05:46.976685 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.976661    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-4dc4b"]
Oct 31 20:05:46.998770 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:46.998742    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"]
Oct 31 20:05:47.001705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:47.001676    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-pbzds"]
Oct 31 20:05:47.018311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:47.018283    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"]
Oct 31 20:05:47.025764 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:47.025737    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-g5qg8"]
Oct 31 20:05:47.046745 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:47.046709    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:05:48.667113 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:48.667078    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" path="/var/lib/kubelet/pods/0592a59d-4a48-432c-acef-7dab11bf3809/volumes"
Oct 31 20:05:48.667792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:48.667770    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" path="/var/lib/kubelet/pods/83a1856b-cd18-4bfb-9a25-68175c3858e6/volumes"
Oct 31 20:05:48.668277 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:48.668257    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" path="/var/lib/kubelet/pods/cdec16c6-aa60-40ff-a953-e3c291e05e24/volumes"
Oct 31 20:05:49.368698 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:49.368637    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"]
Oct 31 20:05:49.397768 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:49.397718    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy" containerID="cri-o://af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e" gracePeriod=30
Oct 31 20:05:49.518632 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:49.518576    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" podUID="b326360d-426f-4051-8d04-7856458d4950" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:49.519066 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:49.519027    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:50.117722 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:50.117678    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"]
Oct 31 20:05:50.921520 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:50.921483    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:50.921751 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:50.921486    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:51.911259 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:51.911224    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"]
Oct 31 20:05:51.950686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:51.950640    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:05:51.982074 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:51.982049    5958 generic.go:356] "Generic (PLEG): container finished" podID="b326360d-426f-4051-8d04-7856458d4950" containerID="b1426ee79149ff92c8e03e69e5daf7470727ebe3eb1e0301de0f0daaac8cda43" exitCode=0
Oct 31 20:05:51.982186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:51.982130    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" event={"ID":"b326360d-426f-4051-8d04-7856458d4950","Type":"ContainerDied","Data":"b1426ee79149ff92c8e03e69e5daf7470727ebe3eb1e0301de0f0daaac8cda43"}
Oct 31 20:05:52.259370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.258894    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 20:05:52.265575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.265512    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 20:05:52.324144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.324102    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d\" (UniqueName: \"kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d\") pod \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") "
Oct 31 20:05:52.324441 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.324157    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume\") pod \"b326360d-426f-4051-8d04-7856458d4950\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") "
Oct 31 20:05:52.324441 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.324195    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume\") pod \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\" (UID: \"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b\") "
Oct 31 20:05:52.324441 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.324240    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8\" (UniqueName: \"kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8\") pod \"b326360d-426f-4051-8d04-7856458d4950\" (UID: \"b326360d-426f-4051-8d04-7856458d4950\") "
Oct 31 20:05:52.325703 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.325669    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume" pod "8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" (UID: "8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:52.325856 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.325739    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume" pod "b326360d-426f-4051-8d04-7856458d4950" (UID: "b326360d-426f-4051-8d04-7856458d4950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:52.327664 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.327499    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8" pod "b326360d-426f-4051-8d04-7856458d4950" (UID: "b326360d-426f-4051-8d04-7856458d4950"). InnerVolumeSpecName "kube-api-access-zjts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:52.328148 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.328117    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d" pod "8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" (UID: "8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b"). InnerVolumeSpecName "kube-api-access-kw96d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:52.426865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.426773    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw96d\" (UniqueName: \"kubernetes.io/projected/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-kube-api-access-kw96d\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:52.426865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.426802    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b326360d-426f-4051-8d04-7856458d4950-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:52.426865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.426812    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:52.426865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.426821    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjts8\" (UniqueName: \"kubernetes.io/projected/b326360d-426f-4051-8d04-7856458d4950-kube-api-access-zjts8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:52.629194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.629138    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:05:52.990588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.990479    5958 generic.go:356] "Generic (PLEG): container finished" podID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerID="f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2" exitCode=0
Oct 31 20:05:52.990588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.990534    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"
Oct 31 20:05:52.991132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.990594    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" event={"ID":"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b","Type":"ContainerDied","Data":"f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2"}
Oct 31 20:05:52.991132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.990654    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr" event={"ID":"8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b","Type":"ContainerDied","Data":"141d4101fead9a3495d85428ea44e3abc30793702e0de54300240f92a3bb57e0"}
Oct 31 20:05:52.991132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.990682    5958 scope.go:122] "RemoveContainer" containerID="f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2"
Oct 31 20:05:52.992086 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.992065    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk" event={"ID":"b326360d-426f-4051-8d04-7856458d4950","Type":"ContainerDied","Data":"5cc4f0bb7a2190693fea817c961a4d7b6862e2759d45a911468c4ef134ff9e16"}
Oct 31 20:05:52.992189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:52.992085    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"
Oct 31 20:05:53.001720 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.001693    5958 scope.go:122] "RemoveContainer" containerID="f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2"
Oct 31 20:05:53.002090 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:53.002067    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2\": container with ID starting with f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2 not found: ID does not exist" containerID="f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2"
Oct 31 20:05:53.002166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.002097    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2"} err="failed to get container status \"f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2\": rpc error: code = NotFound desc = could not find container \"f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2\": container with ID starting with f3e688ae493daff6f9204563037dd550a01cc5c568d30ba5b5f46ff200c447d2 not found: ID does not exist"
Oct 31 20:05:53.002166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.002114    5958 scope.go:122] "RemoveContainer" containerID="b1426ee79149ff92c8e03e69e5daf7470727ebe3eb1e0301de0f0daaac8cda43"
Oct 31 20:05:53.013417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.013397    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"]
Oct 31 20:05:53.020975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.020953    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e1d060-78dc6c4c7f-fv2mr"]
Oct 31 20:05:53.040581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.040552    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"]
Oct 31 20:05:53.053753 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:53.053730    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e1d060-7dfc4c5fc5-l25gk"]
Oct 31 20:05:54.675757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.675679    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" path="/var/lib/kubelet/pods/8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b/volumes"
Oct 31 20:05:54.676303 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.676281    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b326360d-426f-4051-8d04-7856458d4950" path="/var/lib/kubelet/pods/b326360d-426f-4051-8d04-7856458d4950/volumes"
Oct 31 20:05:54.818599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.818568    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 20:05:54.822440 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.822420    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 20:05:54.965007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.964918    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.964992    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965264 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965011    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965264 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965050    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965264 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965157    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume\") pod \"f4aba490-0fe0-4097-9c2e-f74c4527706a\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") "
Oct 31 20:05:54.965264 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965218    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965500 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965263    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965500 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965324    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965500 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965373    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965500 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965418    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy\") pod \"c8e5a527-69f8-4d09-b89b-55d576637e7d\" (UID: \"c8e5a527-69f8-4d09-b89b-55d576637e7d\") "
Oct 31 20:05:54.965500 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965464    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb\" (UniqueName: \"kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb\") pod \"f4aba490-0fe0-4097-9c2e-f74c4527706a\" (UID: \"f4aba490-0fe0-4097-9c2e-f74c4527706a\") "
Oct 31 20:05:54.965799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965629    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:54.965990 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.965961    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:54.966161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.966123    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:54.966161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.966149    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:54.966161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.966149    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:54.966504 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.966378    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:54.966753 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.966641    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume" pod "f4aba490-0fe0-4097-9c2e-f74c4527706a" (UID: "f4aba490-0fe0-4097-9c2e-f74c4527706a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:54.967010 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.966962    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:54.968688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.968662    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:54.968774 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.968672    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:54.969087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.969066    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb" pod "f4aba490-0fe0-4097-9c2e-f74c4527706a" (UID: "f4aba490-0fe0-4097-9c2e-f74c4527706a"). InnerVolumeSpecName "kube-api-access-prhwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:54.970628 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.970600    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:54.970819 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.970799    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n" pod "c8e5a527-69f8-4d09-b89b-55d576637e7d" (UID: "c8e5a527-69f8-4d09-b89b-55d576637e7d"). InnerVolumeSpecName "kube-api-access-vgm8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:54.984803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:54.984778    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 20:05:55.013512 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.013472    5958 generic.go:356] "Generic (PLEG): container finished" podID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerID="002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325" exitCode=0
Oct 31 20:05:55.013755 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.013532    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"
Oct 31 20:05:55.013755 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.013592    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" event={"ID":"f4aba490-0fe0-4097-9c2e-f74c4527706a","Type":"ContainerDied","Data":"002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325"}
Oct 31 20:05:55.013755 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.013652    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7" event={"ID":"f4aba490-0fe0-4097-9c2e-f74c4527706a","Type":"ContainerDied","Data":"0256f021eb63cdc024fdf82c8af01bbbc4bce15424ce7e21a7f40eca7f599cce"}
Oct 31 20:05:55.013755 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.013685    5958 scope.go:122] "RemoveContainer" containerID="002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325"
Oct 31 20:05:55.015732 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.015709    5958 generic.go:356] "Generic (PLEG): container finished" podID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerID="af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e" exitCode=0
Oct 31 20:05:55.015857 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.015827    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"
Oct 31 20:05:55.015944 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.015854    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" event={"ID":"c8e5a527-69f8-4d09-b89b-55d576637e7d","Type":"ContainerDied","Data":"af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e"}
Oct 31 20:05:55.015944 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.015900    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb" event={"ID":"c8e5a527-69f8-4d09-b89b-55d576637e7d","Type":"ContainerDied","Data":"0e490cbfa52e66263c88136f090e1cfba86ae5747a736e2b4754806af8ed8283"}
Oct 31 20:05:55.034043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.034002    5958 generic.go:356] "Generic (PLEG): container finished" podID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerID="9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a" exitCode=0
Oct 31 20:05:55.034205 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.034053    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" event={"ID":"afd6fe3c-12bb-44c2-bd63-5a35ccd15162","Type":"ContainerDied","Data":"9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a"}
Oct 31 20:05:55.034205 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.034100    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj" event={"ID":"afd6fe3c-12bb-44c2-bd63-5a35ccd15162","Type":"ContainerDied","Data":"7d67eb87243bad24f5ed4c2a57acb9f60389f907e033bd23c26168aed71a95be"}
Oct 31 20:05:55.034205 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.034176    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"
Oct 31 20:05:55.036624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.036603    5958 scope.go:122] "RemoveContainer" containerID="002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325"
Oct 31 20:05:55.037142 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:55.037094    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325\": container with ID starting with 002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325 not found: ID does not exist" containerID="002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325"
Oct 31 20:05:55.037278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.037134    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325"} err="failed to get container status \"002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325\": rpc error: code = NotFound desc = could not find container \"002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325\": container with ID starting with 002cbb6b2fee9d70928c4b4b0982d35f27de94284a76785dca33330d6f694325 not found: ID does not exist"
Oct 31 20:05:55.037278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.037158    5958 scope.go:122] "RemoveContainer" containerID="af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e"
Oct 31 20:05:55.052298 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.052274    5958 scope.go:122] "RemoveContainer" containerID="af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e"
Oct 31 20:05:55.052868 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:55.052721    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e\": container with ID starting with af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e not found: ID does not exist" containerID="af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e"
Oct 31 20:05:55.052868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.052759    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e"} err="failed to get container status \"af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e\": rpc error: code = NotFound desc = could not find container \"af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e\": container with ID starting with af3d6ca4665d7cf9f4563c7faf883111d68cd52ea65c55af38cd0e5af880dd5e not found: ID does not exist"
Oct 31 20:05:55.052868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.052780    5958 scope.go:122] "RemoveContainer" containerID="9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a"
Oct 31 20:05:55.064767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.064739    5958 scope.go:122] "RemoveContainer" containerID="9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a"
Oct 31 20:05:55.065107 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:05:55.065081    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a\": container with ID starting with 9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a not found: ID does not exist" containerID="9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a"
Oct 31 20:05:55.065200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.065116    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a"} err="failed to get container status \"9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a\": rpc error: code = NotFound desc = could not find container \"9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a\": container with ID starting with 9027d9b8a7b53aa02fe0db60a45950d39246e9f3532861ae494f09b648fa381a not found: ID does not exist"
Oct 31 20:05:55.067561 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.067522    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r\" (UniqueName: \"kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r\") pod \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") "
Oct 31 20:05:55.067769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.067752    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume\") pod \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\" (UID: \"afd6fe3c-12bb-44c2-bd63-5a35ccd15162\") "
Oct 31 20:05:55.068107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068085    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume" pod "afd6fe3c-12bb-44c2-bd63-5a35ccd15162" (UID: "afd6fe3c-12bb-44c2-bd63-5a35ccd15162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:55.068194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068146    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068165    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgm8n\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-kube-api-access-vgm8n\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068176    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068185    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prhwb\" (UniqueName: \"kubernetes.io/projected/f4aba490-0fe0-4097-9c2e-f74c4527706a-kube-api-access-prhwb\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068194 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068193    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068202    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8e5a527-69f8-4d09-b89b-55d576637e7d-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068210    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8e5a527-69f8-4d09-b89b-55d576637e7d-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068221    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8e5a527-69f8-4d09-b89b-55d576637e7d-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.068482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.068230    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4aba490-0fe0-4097-9c2e-f74c4527706a-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.069779 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.069758    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r" pod "afd6fe3c-12bb-44c2-bd63-5a35ccd15162" (UID: "afd6fe3c-12bb-44c2-bd63-5a35ccd15162"). InnerVolumeSpecName "kube-api-access-snf5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:55.077628 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.077601    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"]
Oct 31 20:05:55.082012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.081991    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-mrrfb"]
Oct 31 20:05:55.095496 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.095467    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"]
Oct 31 20:05:55.100041 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.100012    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ae2e60-67c555dfb9-t8bs7"]
Oct 31 20:05:55.118603 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.118563    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerName="payload-pre-processing" containerID="cri-o://671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d" gracePeriod=25
Oct 31 20:05:55.169981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.169946    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.169981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.169978    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snf5r\" (UniqueName: \"kubernetes.io/projected/afd6fe3c-12bb-44c2-bd63-5a35ccd15162-kube-api-access-snf5r\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.361505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.361476    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"]
Oct 31 20:05:55.370122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:55.370095    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ae2e60-68646c459f-7mfvj"]
Oct 31 20:05:56.666748 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:56.666714    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" path="/var/lib/kubelet/pods/afd6fe3c-12bb-44c2-bd63-5a35ccd15162/volumes"
Oct 31 20:05:56.667232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:56.667217    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" path="/var/lib/kubelet/pods/c8e5a527-69f8-4d09-b89b-55d576637e7d/volumes"
Oct 31 20:05:56.667608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:56.667595    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" path="/var/lib/kubelet/pods/f4aba490-0fe0-4097-9c2e-f74c4527706a/volumes"
Oct 31 20:05:56.912192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:56.912144    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" podUID="799e4486-7297-46e5-8b27-30218f000e80" containerName="payload-pre-processing" containerID="cri-o://845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274" gracePeriod=25
Oct 31 20:05:57.233017 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:57.232979    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"]
Oct 31 20:05:57.291638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:57.291598    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"]
Oct 31 20:05:59.600462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:59.600426    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"]
Oct 31 20:05:59.600984 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:59.600668    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj" podUID="a5d69ddb-4f71-45e8-8baf-b676887a2386" containerName="maas-api" containerID="cri-o://e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef" gracePeriod=30
Oct 31 20:05:59.666129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:59.666090    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main" containerID="cri-o://6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e" gracePeriod=45
Oct 31 20:05:59.672726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:59.672685    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main" containerID="cri-o://6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811" gracePeriod=45
Oct 31 20:05:59.989977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:05:59.989949    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 20:06:00.008922 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.008887    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:06:00.063600 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.063498    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 20:06:00.088624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.088245    5958 generic.go:356] "Generic (PLEG): container finished" podID="1bc33907-ee49-442f-a003-21f6425b4897" containerID="6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e" exitCode=0
Oct 31 20:06:00.088624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.088329    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" event={"ID":"1bc33907-ee49-442f-a003-21f6425b4897","Type":"ContainerDied","Data":"6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e"}
Oct 31 20:06:00.088624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.088358    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg" event={"ID":"1bc33907-ee49-442f-a003-21f6425b4897","Type":"ContainerDied","Data":"e2b41053a1d386cab89e7a384ee400b10adbfd7dccc89acdf577fae421988fdc"}
Oct 31 20:06:00.088624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.088375    5958 scope.go:122] "RemoveContainer" containerID="6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e"
Oct 31 20:06:00.088624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.088482    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"
Oct 31 20:06:00.092416 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.092262    5958 generic.go:356] "Generic (PLEG): container finished" podID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerID="6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811" exitCode=0
Oct 31 20:06:00.092967 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.092362    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"
Oct 31 20:06:00.092967 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.092381    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" event={"ID":"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0","Type":"ContainerDied","Data":"6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811"}
Oct 31 20:06:00.092967 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.092942    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd" event={"ID":"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0","Type":"ContainerDied","Data":"fa25a03532e3c4e6b67b9651b5a8e772e7767a1b35acbfd94a1d149c3bbe34d1"}
Oct 31 20:06:00.094801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.094510    5958 generic.go:356] "Generic (PLEG): container finished" podID="a5d69ddb-4f71-45e8-8baf-b676887a2386" containerID="e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef" exitCode=0
Oct 31 20:06:00.094801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.094607    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj" event={"ID":"a5d69ddb-4f71-45e8-8baf-b676887a2386","Type":"ContainerDied","Data":"e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef"}
Oct 31 20:06:00.094801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.094645    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj" event={"ID":"a5d69ddb-4f71-45e8-8baf-b676887a2386","Type":"ContainerDied","Data":"87fe4c259afd82874fbc185cdd44021fa9bbdd7e6e1b26d73ea57dd9570c04b8"}
Oct 31 20:06:00.094801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.094701    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"
Oct 31 20:06:00.111251 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.110951    5958 scope.go:122] "RemoveContainer" containerID="6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e"
Oct 31 20:06:00.111516 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:00.111374    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e\": container with ID starting with 6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e not found: ID does not exist" containerID="6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e"
Oct 31 20:06:00.111516 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.111413    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e"} err="failed to get container status \"6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e\": rpc error: code = NotFound desc = could not find container \"6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e\": container with ID starting with 6fd4e791992d704edbd0f2663b663f44460efbdea85ea15fb094cbe688c5b90e not found: ID does not exist"
Oct 31 20:06:00.111516 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.111436    5958 scope.go:122] "RemoveContainer" containerID="6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811"
Oct 31 20:06:00.124323 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.124294    5958 scope.go:122] "RemoveContainer" containerID="6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811"
Oct 31 20:06:00.127602 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:00.124778    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811\": container with ID starting with 6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811 not found: ID does not exist" containerID="6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811"
Oct 31 20:06:00.127602 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.124805    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811"} err="failed to get container status \"6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811\": rpc error: code = NotFound desc = could not find container \"6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811\": container with ID starting with 6f1aa6696046e84a1432c08cd651916e7b914b5aa6ab6cad1d6b94566cac1811 not found: ID does not exist"
Oct 31 20:06:00.127602 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.124827    5958 scope.go:122] "RemoveContainer" containerID="e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef"
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143444    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home\") pod \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143534    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs\") pod \"a5d69ddb-4f71-45e8-8baf-b676887a2386\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143606    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache\") pod \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143671    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm\") pod \"1bc33907-ee49-442f-a003-21f6425b4897\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143782    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm\") pod \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143809    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls\") pod \"a5d69ddb-4f71-45e8-8baf-b676887a2386\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache\") pod \"1bc33907-ee49-442f-a003-21f6425b4897\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143911    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs\") pod \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143948    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc\" (UniqueName: \"kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc\") pod \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.143990    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca\") pod \"a5d69ddb-4f71-45e8-8baf-b676887a2386\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144037    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home\") pod \"1bc33907-ee49-442f-a003-21f6425b4897\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144116    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w\" (UniqueName: \"kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w\") pod \"a5d69ddb-4f71-45e8-8baf-b676887a2386\" (UID: \"a5d69ddb-4f71-45e8-8baf-b676887a2386\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144145    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir\") pod \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\" (UID: \"5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144168    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home" pod "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" (UID: "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144186    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir\") pod \"1bc33907-ee49-442f-a003-21f6425b4897\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144216    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m\" (UniqueName: \"kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m\") pod \"1bc33907-ee49-442f-a003-21f6425b4897\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") "
Oct 31 20:06:00.145079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.144268    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs\") pod \"1bc33907-ee49-442f-a003-21f6425b4897\" (UID: \"1bc33907-ee49-442f-a003-21f6425b4897\") "
Oct 31 20:06:00.147705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.146504    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.147705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.146499    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home" pod "1bc33907-ee49-442f-a003-21f6425b4897" (UID: "1bc33907-ee49-442f-a003-21f6425b4897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.147705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.146687    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir" pod "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" (UID: "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.147705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.146814    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache" pod "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" (UID: "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.150884 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.148440    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir" pod "1bc33907-ee49-442f-a003-21f6425b4897" (UID: "1bc33907-ee49-442f-a003-21f6425b4897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.150884 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.150277    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc" pod "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" (UID: "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0"). InnerVolumeSpecName "kube-api-access-w4dfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:00.150884 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.148390    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache" pod "1bc33907-ee49-442f-a003-21f6425b4897" (UID: "1bc33907-ee49-442f-a003-21f6425b4897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.151383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.150652    5958 scope.go:122] "RemoveContainer" containerID="e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef"
Oct 31 20:06:00.151383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.150859    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca" pod "a5d69ddb-4f71-45e8-8baf-b676887a2386" (UID: "a5d69ddb-4f71-45e8-8baf-b676887a2386"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:00.152324 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:00.151863    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef\": container with ID starting with e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef not found: ID does not exist" containerID="e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef"
Oct 31 20:06:00.152324 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.151922    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef"} err="failed to get container status \"e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef\": rpc error: code = NotFound desc = could not find container \"e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef\": container with ID starting with e49bcd5e7a0e949e5cadc966fbaf71e1ad1d4f6937a41aabcd166578217a12ef not found: ID does not exist"
Oct 31 20:06:00.153971 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.153941    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs" pod "a5d69ddb-4f71-45e8-8baf-b676887a2386" (UID: "a5d69ddb-4f71-45e8-8baf-b676887a2386"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:00.154384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.154364    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs" pod "1bc33907-ee49-442f-a003-21f6425b4897" (UID: "1bc33907-ee49-442f-a003-21f6425b4897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:00.154621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.154581    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w" pod "a5d69ddb-4f71-45e8-8baf-b676887a2386" (UID: "a5d69ddb-4f71-45e8-8baf-b676887a2386"). InnerVolumeSpecName "kube-api-access-v5r9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:00.154621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.154610    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls" pod "a5d69ddb-4f71-45e8-8baf-b676887a2386" (UID: "a5d69ddb-4f71-45e8-8baf-b676887a2386"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:00.154789 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.154769    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm" pod "1bc33907-ee49-442f-a003-21f6425b4897" (UID: "1bc33907-ee49-442f-a003-21f6425b4897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.154829 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.154809    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs" pod "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" (UID: "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:00.154869 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.154852    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m" pod "1bc33907-ee49-442f-a003-21f6425b4897" (UID: "1bc33907-ee49-442f-a003-21f6425b4897"). InnerVolumeSpecName "kube-api-access-ptb8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:00.156061 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.156034    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm" pod "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" (UID: "5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:00.248200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248160    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248198    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248209    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dfc\" (UniqueName: \"kubernetes.io/projected/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-kube-api-access-w4dfc\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248218    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5d69ddb-4f71-45e8-8baf-b676887a2386-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248228    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248236    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5r9w\" (UniqueName: \"kubernetes.io/projected/a5d69ddb-4f71-45e8-8baf-b676887a2386-kube-api-access-v5r9w\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248246    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248255    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248263    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptb8m\" (UniqueName: \"kubernetes.io/projected/1bc33907-ee49-442f-a003-21f6425b4897-kube-api-access-ptb8m\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248271    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc33907-ee49-442f-a003-21f6425b4897-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248280    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248290    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248298    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc33907-ee49-442f-a003-21f6425b4897-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248305    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.248482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.248313    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5d69ddb-4f71-45e8-8baf-b676887a2386-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.452834 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.452771    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"]
Oct 31 20:06:00.460297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.460264    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-w6-8730d5-kserve-f695d54f4-rdgkg"]
Oct 31 20:06:00.499560 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.499518    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"]
Oct 31 20:06:00.503456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.503425    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/premium-w6-8730d5-kserve-5f68b56bc9-2cljd"]
Oct 31 20:06:00.527209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.527179    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"]
Oct 31 20:06:00.531971 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.531946    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c173d8-599c85f758-6b5jj"]
Oct 31 20:06:00.665087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.665016    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc33907-ee49-442f-a003-21f6425b4897" path="/var/lib/kubelet/pods/1bc33907-ee49-442f-a003-21f6425b4897/volumes"
Oct 31 20:06:00.665472 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.665458    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" path="/var/lib/kubelet/pods/5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0/volumes"
Oct 31 20:06:00.665926 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:00.665911    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d69ddb-4f71-45e8-8baf-b676887a2386" path="/var/lib/kubelet/pods/a5d69ddb-4f71-45e8-8baf-b676887a2386/volumes"
Oct 31 20:06:01.563321 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.563251    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main" containerID="cri-o://5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53" gracePeriod=45
Oct 31 20:06:01.865754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.865727    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 20:06:01.968857 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.968818    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir\") pod \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") "
Oct 31 20:06:01.969039 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.968936    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache\") pod \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") "
Oct 31 20:06:01.969039 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.968961    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm\") pod \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") "
Oct 31 20:06:01.969039 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969024    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs\" (UniqueName: \"kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs\") pod \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") "
Oct 31 20:06:01.969219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969045    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home\") pod \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") "
Oct 31 20:06:01.969219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969081    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2\" (UniqueName: \"kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2\") pod \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\" (UID: \"919fbf1d-07fd-46e3-896b-b82e7b9985bb\") "
Oct 31 20:06:01.969219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969080    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir" pod "919fbf1d-07fd-46e3-896b-b82e7b9985bb" (UID: "919fbf1d-07fd-46e3-896b-b82e7b9985bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:01.969359 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969248    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache" pod "919fbf1d-07fd-46e3-896b-b82e7b9985bb" (UID: "919fbf1d-07fd-46e3-896b-b82e7b9985bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:01.969359 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969299    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home" pod "919fbf1d-07fd-46e3-896b-b82e7b9985bb" (UID: "919fbf1d-07fd-46e3-896b-b82e7b9985bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:01.969604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969585    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:01.969604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969604    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:01.969747 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.969613    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:01.971761 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.971733    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm" pod "919fbf1d-07fd-46e3-896b-b82e7b9985bb" (UID: "919fbf1d-07fd-46e3-896b-b82e7b9985bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:01.971874 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.971755    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2" pod "919fbf1d-07fd-46e3-896b-b82e7b9985bb" (UID: "919fbf1d-07fd-46e3-896b-b82e7b9985bb"). InnerVolumeSpecName "kube-api-access-dktr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:01.972011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:01.971993    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs" pod "919fbf1d-07fd-46e3-896b-b82e7b9985bb" (UID: "919fbf1d-07fd-46e3-896b-b82e7b9985bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:02.071023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.070989    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919fbf1d-07fd-46e3-896b-b82e7b9985bb-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.071023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.071023    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919fbf1d-07fd-46e3-896b-b82e7b9985bb-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.071241 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.071036    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktr2\" (UniqueName: \"kubernetes.io/projected/919fbf1d-07fd-46e3-896b-b82e7b9985bb-kube-api-access-dktr2\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.113047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.112958    5958 generic.go:356] "Generic (PLEG): container finished" podID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerID="5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53" exitCode=0
Oct 31 20:06:02.113047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.113010    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"
Oct 31 20:06:02.113290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.113043    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" event={"ID":"919fbf1d-07fd-46e3-896b-b82e7b9985bb","Type":"ContainerDied","Data":"5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53"}
Oct 31 20:06:02.113290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.113086    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z" event={"ID":"919fbf1d-07fd-46e3-896b-b82e7b9985bb","Type":"ContainerDied","Data":"f88b4050981fc96f0461b4780667b558d4078591a3739d7c847a0f7eea2c2f35"}
Oct 31 20:06:02.113290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.113104    5958 scope.go:122] "RemoveContainer" containerID="5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53"
Oct 31 20:06:02.124189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.124168    5958 scope.go:122] "RemoveContainer" containerID="5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53"
Oct 31 20:06:02.124520 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:02.124496    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53\": container with ID starting with 5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53 not found: ID does not exist" containerID="5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53"
Oct 31 20:06:02.124603 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.124528    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53"} err="failed to get container status \"5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53\": rpc error: code = NotFound desc = could not find container \"5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53\": container with ID starting with 5f7f2d2af7ba46d22254bd0d72db33d4262d6131dc1a52b84bf87e0f84fe9f53 not found: ID does not exist"
Oct 31 20:06:02.143151 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.143119    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"]
Oct 31 20:06:02.145931 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.145904    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-2-w2-aeb292-kserve-d995cd58f-96k4z"]
Oct 31 20:06:02.233778 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.233733    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn" podUID="84222e00-b33d-4828-839e-b4eefe6c5c64" containerName="payload-processing" containerID="cri-o://4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13" gracePeriod=25
Oct 31 20:06:02.292052 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.291998    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm" podUID="3af08b8b-8eea-409a-8573-3468870673c9" containerName="payload-pre-processing" containerID="cri-o://3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14" gracePeriod=25
Oct 31 20:06:02.570289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.570256    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:06:02.613167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.613141    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:06:02.666577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.666475    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" path="/var/lib/kubelet/pods/919fbf1d-07fd-46e3-896b-b82e7b9985bb/volumes"
Oct 31 20:06:02.679706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.679672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume\") pod \"84222e00-b33d-4828-839e-b4eefe6c5c64\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") "
Oct 31 20:06:02.679901 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.679819    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr\" (UniqueName: \"kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr\") pod \"84222e00-b33d-4828-839e-b4eefe6c5c64\" (UID: \"84222e00-b33d-4828-839e-b4eefe6c5c64\") "
Oct 31 20:06:02.680152 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.680123    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume" pod "84222e00-b33d-4828-839e-b4eefe6c5c64" (UID: "84222e00-b33d-4828-839e-b4eefe6c5c64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:02.682257 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.682232    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr" pod "84222e00-b33d-4828-839e-b4eefe6c5c64" (UID: "84222e00-b33d-4828-839e-b4eefe6c5c64"). InnerVolumeSpecName "kube-api-access-c56xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:02.780873 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.780833    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume\") pod \"3af08b8b-8eea-409a-8573-3468870673c9\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") "
Oct 31 20:06:02.781090 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.780947    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49\" (UniqueName: \"kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49\") pod \"3af08b8b-8eea-409a-8573-3468870673c9\" (UID: \"3af08b8b-8eea-409a-8573-3468870673c9\") "
Oct 31 20:06:02.781729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.781281    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume" pod "3af08b8b-8eea-409a-8573-3468870673c9" (UID: "3af08b8b-8eea-409a-8573-3468870673c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:02.782395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.782367    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c56xr\" (UniqueName: \"kubernetes.io/projected/84222e00-b33d-4828-839e-b4eefe6c5c64-kube-api-access-c56xr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.782482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.782401    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af08b8b-8eea-409a-8573-3468870673c9-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.782482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.782418    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84222e00-b33d-4828-839e-b4eefe6c5c64-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.783602 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.783579    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49" pod "3af08b8b-8eea-409a-8573-3468870673c9" (UID: "3af08b8b-8eea-409a-8573-3468870673c9"). InnerVolumeSpecName "kube-api-access-bpw49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:02.884137 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:02.884100    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpw49\" (UniqueName: \"kubernetes.io/projected/3af08b8b-8eea-409a-8573-3468870673c9-kube-api-access-bpw49\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:03.121461 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.121430    5958 generic.go:356] "Generic (PLEG): container finished" podID="3af08b8b-8eea-409a-8573-3468870673c9" containerID="3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14" exitCode=0
Oct 31 20:06:03.121661 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.121475    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"
Oct 31 20:06:03.121661 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.121506    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm" event={"ID":"3af08b8b-8eea-409a-8573-3468870673c9","Type":"ContainerDied","Data":"3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14"}
Oct 31 20:06:03.121661 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.121580    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm" event={"ID":"3af08b8b-8eea-409a-8573-3468870673c9","Type":"ContainerDied","Data":"a885592a9ec58588e74b3d0fcd92d4f80075d2b11470585a86ff7e7debc879f2"}
Oct 31 20:06:03.121661 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.121614    5958 scope.go:122] "RemoveContainer" containerID="3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14"
Oct 31 20:06:03.123081 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.123058    5958 generic.go:356] "Generic (PLEG): container finished" podID="84222e00-b33d-4828-839e-b4eefe6c5c64" containerID="4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13" exitCode=0
Oct 31 20:06:03.123209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.123088    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn" event={"ID":"84222e00-b33d-4828-839e-b4eefe6c5c64","Type":"ContainerDied","Data":"4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13"}
Oct 31 20:06:03.123209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.123109    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"
Oct 31 20:06:03.123209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.123140    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn" event={"ID":"84222e00-b33d-4828-839e-b4eefe6c5c64","Type":"ContainerDied","Data":"f36893793fd0ccf6697d8b83e7b7814a7eba403f01913e2e48251fccab2a5ab4"}
Oct 31 20:06:03.133610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.133589    5958 scope.go:122] "RemoveContainer" containerID="3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14"
Oct 31 20:06:03.133928 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:03.133906    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14\": container with ID starting with 3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14 not found: ID does not exist" containerID="3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14"
Oct 31 20:06:03.134016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.133937    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14"} err="failed to get container status \"3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14\": rpc error: code = NotFound desc = could not find container \"3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14\": container with ID starting with 3d85166395b872ea85d4a89433973462e4e6b68cd573421c526d3629075c8f14 not found: ID does not exist"
Oct 31 20:06:03.134016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.133959    5958 scope.go:122] "RemoveContainer" containerID="4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13"
Oct 31 20:06:03.159694 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.158215    5958 scope.go:122] "RemoveContainer" containerID="4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13"
Oct 31 20:06:03.159694 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:03.158663    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13\": container with ID starting with 4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13 not found: ID does not exist" containerID="4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13"
Oct 31 20:06:03.159694 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.158704    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13"} err="failed to get container status \"4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13\": rpc error: code = NotFound desc = could not find container \"4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13\": container with ID starting with 4a7e775e671e729a5195f60d6f4cbecef58701ca4670eed69b2f64d0581a6c13 not found: ID does not exist"
Oct 31 20:06:03.159694 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.159621    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"]
Oct 31 20:06:03.162476 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.162452    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-79c665fbc9-ft6fm"]
Oct 31 20:06:03.197735 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.197697    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"]
Oct 31 20:06:03.211412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.211377    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-6db886499c-vfjqn"]
Oct 31 20:06:03.436598 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.436498    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:06:03.861602 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.861505    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"]
Oct 31 20:06:03.861807 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:03.861776    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8" podUID="9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" containerName="maas-api" containerID="cri-o://c6641beba483562848c7282ab287db56d5b65ba82608df53784bf36f41617ffb" gracePeriod=30
Oct 31 20:06:04.137063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.135873    5958 generic.go:356] "Generic (PLEG): container finished" podID="9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" containerID="c6641beba483562848c7282ab287db56d5b65ba82608df53784bf36f41617ffb" exitCode=0
Oct 31 20:06:04.137063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.135997    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8" event={"ID":"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94","Type":"ContainerDied","Data":"c6641beba483562848c7282ab287db56d5b65ba82608df53784bf36f41617ffb"}
Oct 31 20:06:04.160377 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.160347    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 20:06:04.299032 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.298997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs\") pod \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") "
Oct 31 20:06:04.299214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.299083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca\") pod \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") "
Oct 31 20:06:04.299214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.299108    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls\") pod \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") "
Oct 31 20:06:04.299214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.299169    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf\" (UniqueName: \"kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf\") pod \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\" (UID: \"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94\") "
Oct 31 20:06:04.299536 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.299500    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca" pod "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" (UID: "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:04.301290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.301266    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls" pod "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" (UID: "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:04.301382 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.301306    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs" pod "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" (UID: "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:04.301382 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.301354    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf" pod "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" (UID: "9f1dbc0d-2dd5-4450-9742-a8e2274e1f94"). InnerVolumeSpecName "kube-api-access-xplkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:04.400314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.400227    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:04.400314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.400261    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:04.400314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.400271    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:04.400314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.400281    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xplkf\" (UniqueName: \"kubernetes.io/projected/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94-kube-api-access-xplkf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:04.444290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.444248    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" podUID="799e4486-7297-46e5-8b27-30218f000e80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:06:04.664686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.664609    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af08b8b-8eea-409a-8573-3468870673c9" path="/var/lib/kubelet/pods/3af08b8b-8eea-409a-8573-3468870673c9/volumes"
Oct 31 20:06:04.665083 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:04.665069    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84222e00-b33d-4828-839e-b4eefe6c5c64" path="/var/lib/kubelet/pods/84222e00-b33d-4828-839e-b4eefe6c5c64/volumes"
Oct 31 20:06:05.144492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:05.144459    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"
Oct 31 20:06:05.144492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:05.144486    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8" event={"ID":"9f1dbc0d-2dd5-4450-9742-a8e2274e1f94","Type":"ContainerDied","Data":"5ab8a62f1f570915a38de64f5b74928a9e6e339dfc401e29cad8294745c862b0"}
Oct 31 20:06:05.145076 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:05.144577    5958 scope.go:122] "RemoveContainer" containerID="c6641beba483562848c7282ab287db56d5b65ba82608df53784bf36f41617ffb"
Oct 31 20:06:05.170257 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:05.170228    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"]
Oct 31 20:06:05.179908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:05.179883    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-aeb292-84f9bcf898-nsjm8"]
Oct 31 20:06:06.664723 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:06.664693    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" path="/var/lib/kubelet/pods/9f1dbc0d-2dd5-4450-9742-a8e2274e1f94/volumes"
Oct 31 20:06:07.540856 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.540822    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"]
Oct 31 20:06:07.541456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541442    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerName="payload-processing"
Oct 31 20:06:07.541501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541456    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerName="payload-processing"
Oct 31 20:06:07.541501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541471    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3af08b8b-8eea-409a-8573-3468870673c9" containerName="payload-pre-processing"
Oct 31 20:06:07.541501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541477    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af08b8b-8eea-409a-8573-3468870673c9" containerName="payload-pre-processing"
Oct 31 20:06:07.541501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541487    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy"
Oct 31 20:06:07.541501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541492    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541505    5958 cpu_manager.go:414] "containerMap: removing container" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerName="payload-pre-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541511    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerName="payload-pre-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541527    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541532    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541556    5958 cpu_manager.go:414] "containerMap: removing container" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541561    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541572    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" containerName="maas-api"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541576    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" containerName="maas-api"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541584    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541588    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541594    5958 cpu_manager.go:414] "containerMap: removing container" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541599    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541615    5958 cpu_manager.go:414] "containerMap: removing container" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541620    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541629    5958 cpu_manager.go:414] "containerMap: removing container" podUID="84222e00-b33d-4828-839e-b4eefe6c5c64" containerName="payload-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541633    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84222e00-b33d-4828-839e-b4eefe6c5c64" containerName="payload-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541639    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541644    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541653    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerName="payload-pre-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541658    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerName="payload-pre-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541670    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b326360d-426f-4051-8d04-7856458d4950" containerName="payload-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541675    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b326360d-426f-4051-8d04-7856458d4950" containerName="payload-processing"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541682    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541686    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541695    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a5d69ddb-4f71-45e8-8baf-b676887a2386" containerName="maas-api"
Oct 31 20:06:07.541743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541699    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d69ddb-4f71-45e8-8baf-b676887a2386" containerName="maas-api"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541773    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b326360d-426f-4051-8d04-7856458d4950" containerName="payload-processing"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541783    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc33907-ee49-442f-a003-21f6425b4897" containerName="main"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541789    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d69ddb-4f71-45e8-8baf-b676887a2386" containerName="maas-api"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541795    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4aba490-0fe0-4097-9c2e-f74c4527706a" containerName="payload-processing"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541800    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af08b8b-8eea-409a-8573-3468870673c9" containerName="payload-pre-processing"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541806    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a847c2db-ec3f-4e41-8140-5e7a9db18309" containerName="istio-proxy"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541812    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd6fe3c-12bb-44c2-bd63-5a35ccd15162" containerName="payload-pre-processing"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541820    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a15b93d-e8a6-4a8e-ba3a-fe9ced40c4e0" containerName="main"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541826    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdec16c6-aa60-40ff-a953-e3c291e05e24" containerName="istio-proxy"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541832    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1dbc0d-2dd5-4450-9742-a8e2274e1f94" containerName="maas-api"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541838    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0592a59d-4a48-432c-acef-7dab11bf3809" containerName="istio-proxy"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541844    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="919fbf1d-07fd-46e3-896b-b82e7b9985bb" containerName="main"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541850    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a1856b-cd18-4bfb-9a25-68175c3858e6" containerName="istio-proxy"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541857    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="84222e00-b33d-4828-839e-b4eefe6c5c64" containerName="payload-processing"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541861    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e5a527-69f8-4d09-b89b-55d576637e7d" containerName="istio-proxy"
Oct 31 20:06:07.542575 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.541866    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8075a7a5-54cd-4a59-bf49-a4a0f8a61e6b" containerName="payload-pre-processing"
Oct 31 20:06:07.547210 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.547186    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.561053 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.561022    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"]
Oct 31 20:06:07.563635 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.563611    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7fe845-dockercfg-ts76p\""
Oct 31 20:06:07.634298 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.634262    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7fe845\""
Oct 31 20:06:07.642245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.642107    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.649428 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.649388    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2nc\" (UniqueName: \"kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.652212 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.652168    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"]
Oct 31 20:06:07.657930 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.657904    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.660514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.660476    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"]
Oct 31 20:06:07.750734 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.750701    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.751235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.750764    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2nc\" (UniqueName: \"kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.751235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.750796    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.751235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.750846    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8f24\" (UniqueName: \"kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24\") pod \"payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.751670 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.751645    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.765092 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.765063    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2nc\" (UniqueName: \"kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc\") pod \"payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.852067 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.851975    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8f24\" (UniqueName: \"kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24\") pod \"payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.852235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.852123    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.852885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.852861    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.865540 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.865516    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:07.866005 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.865980    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8f24\" (UniqueName: \"kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24\") pod \"payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:07.988317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:07.987777    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:08.152447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:08.152418    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"]
Oct 31 20:06:08.154500 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:06:08.154471    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d8b407_edfa_4b0b_925d_ecaf2d014038.slice/crio-f378bcaed1458b063a005a22b7a4d75da17ca173a6c6058fa65217337c942474 WatchSource:0}: Error finding container f378bcaed1458b063a005a22b7a4d75da17ca173a6c6058fa65217337c942474: Status 404 returned error can't find the container with id f378bcaed1458b063a005a22b7a4d75da17ca173a6c6058fa65217337c942474
Oct 31 20:06:08.168001 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:08.167974    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2" event={"ID":"79d8b407-edfa-4b0b-925d-ecaf2d014038","Type":"ContainerStarted","Data":"f378bcaed1458b063a005a22b7a4d75da17ca173a6c6058fa65217337c942474"}
Oct 31 20:06:08.276859 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:08.276818    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"]
Oct 31 20:06:08.277222 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:06:08.277193    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2738d1_4587_4dfd_8ea4_2542d0d9b842.slice/crio-1b5dd8d204da76b28f9410b554571d2bc69da79ceb45a3a08b6bf3285486a878 WatchSource:0}: Error finding container 1b5dd8d204da76b28f9410b554571d2bc69da79ceb45a3a08b6bf3285486a878: Status 404 returned error can't find the container with id 1b5dd8d204da76b28f9410b554571d2bc69da79ceb45a3a08b6bf3285486a878
Oct 31 20:06:09.178436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.178395    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2" event={"ID":"79d8b407-edfa-4b0b-925d-ecaf2d014038","Type":"ContainerStarted","Data":"f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893"}
Oct 31 20:06:09.178989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.178614    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:09.180172 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.180142    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs" event={"ID":"5a2738d1-4587-4dfd-8ea4-2542d0d9b842","Type":"ContainerStarted","Data":"52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045"}
Oct 31 20:06:09.180307 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.180194    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs" event={"ID":"5a2738d1-4587-4dfd-8ea4-2542d0d9b842","Type":"ContainerStarted","Data":"1b5dd8d204da76b28f9410b554571d2bc69da79ceb45a3a08b6bf3285486a878"}
Oct 31 20:06:09.180307 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.180225    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:09.201577 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.201504    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2" podStartSLOduration=2.201485146 podStartE2EDuration="2.201485146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:09.199930628 +0000 UTC m=+2762.661305245" watchObservedRunningTime="2026-10-31 20:06:09.201485146 +0000 UTC m=+2762.662859763"
Oct 31 20:06:09.226603 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.226508    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs" podStartSLOduration=2.226489748 podStartE2EDuration="2.226489748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:09.224823945 +0000 UTC m=+2762.686198561" watchObservedRunningTime="2026-10-31 20:06:09.226489748 +0000 UTC m=+2762.687864364"
Oct 31 20:06:09.885208 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:09.885173    5958 scope.go:122] "RemoveContainer" containerID="8cb6af90701cc60deab8bed7aa79e8501ed635251963b7b8fc2187e14a7c147f"
Oct 31 20:06:13.288486 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:13.288448    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:06:13.437020 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:13.436982    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:06:14.444346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:14.444307    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" podUID="799e4486-7297-46e5-8b27-30218f000e80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:06:15.398347 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.398298    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 20:06:15.452686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.452652    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume\") pod \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") "
Oct 31 20:06:15.453266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.452724    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd\" (UniqueName: \"kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd\") pod \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\" (UID: \"3a3a29c0-4c9f-4740-b379-d0d54dbb818e\") "
Oct 31 20:06:15.453266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.453082    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume" pod "3a3a29c0-4c9f-4740-b379-d0d54dbb818e" (UID: "3a3a29c0-4c9f-4740-b379-d0d54dbb818e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:15.453399 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.453346    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:15.455183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.455154    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd" pod "3a3a29c0-4c9f-4740-b379-d0d54dbb818e" (UID: "3a3a29c0-4c9f-4740-b379-d0d54dbb818e"). InnerVolumeSpecName "kube-api-access-csvjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:15.554588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:15.554524    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csvjd\" (UniqueName: \"kubernetes.io/projected/3a3a29c0-4c9f-4740-b379-d0d54dbb818e-kube-api-access-csvjd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:16.237015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.236982    5958 generic.go:356] "Generic (PLEG): container finished" podID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerID="671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d" exitCode=0
Oct 31 20:06:16.237207 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.237034    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"
Oct 31 20:06:16.237207 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.237041    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" event={"ID":"3a3a29c0-4c9f-4740-b379-d0d54dbb818e","Type":"ContainerDied","Data":"671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d"}
Oct 31 20:06:16.237207 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.237084    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5" event={"ID":"3a3a29c0-4c9f-4740-b379-d0d54dbb818e","Type":"ContainerDied","Data":"481cbd6599a9c7405f2ab7144918f33984f39eea0c4f7a0be09182b61684bec7"}
Oct 31 20:06:16.237207 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.237112    5958 scope.go:122] "RemoveContainer" containerID="671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d"
Oct 31 20:06:16.248604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.248436    5958 scope.go:122] "RemoveContainer" containerID="671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d"
Oct 31 20:06:16.248772 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:16.248753    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d\": container with ID starting with 671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d not found: ID does not exist" containerID="671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d"
Oct 31 20:06:16.248815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.248780    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d"} err="failed to get container status \"671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d\": rpc error: code = NotFound desc = could not find container \"671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d\": container with ID starting with 671699d2c3e4954fd59905f9ba6d238f6c2d6c952b8c670ca3fdef193d8ca25d not found: ID does not exist"
Oct 31 20:06:16.264665 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.264636    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"]
Oct 31 20:06:16.272466 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.272439    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8730d5-569cd886cf-shqb5"]
Oct 31 20:06:16.663783 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:16.663748    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" path="/var/lib/kubelet/pods/3a3a29c0-4c9f-4740-b379-d0d54dbb818e/volumes"
Oct 31 20:06:17.051962 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:17.051928    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:06:17.187629 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.187561    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 20:06:17.246759 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.246718    5958 generic.go:356] "Generic (PLEG): container finished" podID="799e4486-7297-46e5-8b27-30218f000e80" containerID="845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274" exitCode=0
Oct 31 20:06:17.247031 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.246776    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"
Oct 31 20:06:17.247031 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.246808    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" event={"ID":"799e4486-7297-46e5-8b27-30218f000e80","Type":"ContainerDied","Data":"845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274"}
Oct 31 20:06:17.247031 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.246871    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt" event={"ID":"799e4486-7297-46e5-8b27-30218f000e80","Type":"ContainerDied","Data":"dbbf191fa2f89b23cc1b4fb128c8fc9152912b230851b05cba3d70e5fb1eb204"}
Oct 31 20:06:17.247031 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.246903    5958 scope.go:122] "RemoveContainer" containerID="845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274"
Oct 31 20:06:17.258774 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.258598    5958 scope.go:122] "RemoveContainer" containerID="845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274"
Oct 31 20:06:17.258948 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:17.258928    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274\": container with ID starting with 845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274 not found: ID does not exist" containerID="845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274"
Oct 31 20:06:17.259004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.258958    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274"} err="failed to get container status \"845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274\": rpc error: code = NotFound desc = could not find container \"845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274\": container with ID starting with 845a55d25e3d893e05f4ec73ecc1ec4bccc1c15f77973589bd6f2f47bfa5d274 not found: ID does not exist"
Oct 31 20:06:17.274365 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.274335    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume\") pod \"799e4486-7297-46e5-8b27-30218f000e80\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") "
Oct 31 20:06:17.274533 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.274497    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96\" (UniqueName: \"kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96\") pod \"799e4486-7297-46e5-8b27-30218f000e80\" (UID: \"799e4486-7297-46e5-8b27-30218f000e80\") "
Oct 31 20:06:17.274768 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.274744    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume" pod "799e4486-7297-46e5-8b27-30218f000e80" (UID: "799e4486-7297-46e5-8b27-30218f000e80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:17.274965 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.274948    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/799e4486-7297-46e5-8b27-30218f000e80-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:17.276576 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.276532    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96" pod "799e4486-7297-46e5-8b27-30218f000e80" (UID: "799e4486-7297-46e5-8b27-30218f000e80"). InnerVolumeSpecName "kube-api-access-xzk96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:17.375774 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.375739    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzk96\" (UniqueName: \"kubernetes.io/projected/799e4486-7297-46e5-8b27-30218f000e80-kube-api-access-xzk96\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:17.580706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.580650    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"]
Oct 31 20:06:17.583073 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:17.583041    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-aeb292-5fbf46df9-w8bqt"]
Oct 31 20:06:18.664247 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:18.664209    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799e4486-7297-46e5-8b27-30218f000e80" path="/var/lib/kubelet/pods/799e4486-7297-46e5-8b27-30218f000e80/volumes"
Oct 31 20:06:20.193108 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:20.193075    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:06:20.193653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:20.193259    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:06:27.250927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:27.250893    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs_5a2738d1-4587-4dfd-8ea4-2542d0d9b842/payload-processing/0.log"
Oct 31 20:06:32.299711 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:32.299662    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:06:39.660666 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:39.660625    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"]
Oct 31 20:06:39.661220 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:39.660858    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb" podUID="5783b36a-c79f-47b7-a407-cbfaf48567c1" containerName="maas-api" containerID="cri-o://6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002" gracePeriod=30
Oct 31 20:06:39.968245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:39.968219    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:06:40.070181 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.070147    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca\") pod \"5783b36a-c79f-47b7-a407-cbfaf48567c1\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") "
Oct 31 20:06:40.070402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.070209    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4\" (UniqueName: \"kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4\") pod \"5783b36a-c79f-47b7-a407-cbfaf48567c1\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") "
Oct 31 20:06:40.070402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.070229    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs\") pod \"5783b36a-c79f-47b7-a407-cbfaf48567c1\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") "
Oct 31 20:06:40.070402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.070311    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls\") pod \"5783b36a-c79f-47b7-a407-cbfaf48567c1\" (UID: \"5783b36a-c79f-47b7-a407-cbfaf48567c1\") "
Oct 31 20:06:40.070673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.070644    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca" pod "5783b36a-c79f-47b7-a407-cbfaf48567c1" (UID: "5783b36a-c79f-47b7-a407-cbfaf48567c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:40.070911 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.070893    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5783b36a-c79f-47b7-a407-cbfaf48567c1-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.072558 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.072510    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs" pod "5783b36a-c79f-47b7-a407-cbfaf48567c1" (UID: "5783b36a-c79f-47b7-a407-cbfaf48567c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:40.072673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.072633    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls" pod "5783b36a-c79f-47b7-a407-cbfaf48567c1" (UID: "5783b36a-c79f-47b7-a407-cbfaf48567c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:40.072673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.072641    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4" pod "5783b36a-c79f-47b7-a407-cbfaf48567c1" (UID: "5783b36a-c79f-47b7-a407-cbfaf48567c1"). InnerVolumeSpecName "kube-api-access-f4fw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:40.177104 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.177063    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4fw4\" (UniqueName: \"kubernetes.io/projected/5783b36a-c79f-47b7-a407-cbfaf48567c1-kube-api-access-f4fw4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.177104 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.177105    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.177394 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.177120    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5783b36a-c79f-47b7-a407-cbfaf48567c1-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.413833 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.413792    5958 generic.go:356] "Generic (PLEG): container finished" podID="5783b36a-c79f-47b7-a407-cbfaf48567c1" containerID="6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002" exitCode=0
Oct 31 20:06:40.414029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.413843    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"
Oct 31 20:06:40.414029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.413870    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb" event={"ID":"5783b36a-c79f-47b7-a407-cbfaf48567c1","Type":"ContainerDied","Data":"6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002"}
Oct 31 20:06:40.414029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.413924    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb" event={"ID":"5783b36a-c79f-47b7-a407-cbfaf48567c1","Type":"ContainerDied","Data":"7880b5cc20c58b2d818354f35906ff02bacb10ed8440624090bd37d17a1f0b3d"}
Oct 31 20:06:40.414029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.413946    5958 scope.go:122] "RemoveContainer" containerID="6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002"
Oct 31 20:06:40.430785 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.430613    5958 scope.go:122] "RemoveContainer" containerID="6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002"
Oct 31 20:06:40.431321 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:40.431292    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002\": container with ID starting with 6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002 not found: ID does not exist" containerID="6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002"
Oct 31 20:06:40.431414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.431338    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002"} err="failed to get container status \"6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002\": rpc error: code = NotFound desc = could not find container \"6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002\": container with ID starting with 6a7847f58670678c1959058db8aef8d3e6e28d2ba6bac3bd1e1db0dd1cc80002 not found: ID does not exist"
Oct 31 20:06:40.447580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.447536    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"]
Oct 31 20:06:40.451733 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.451707    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f28d48-54f58b9cb-ck2mb"]
Oct 31 20:06:40.668108 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:40.668017    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5783b36a-c79f-47b7-a407-cbfaf48567c1" path="/var/lib/kubelet/pods/5783b36a-c79f-47b7-a407-cbfaf48567c1/volumes"
Oct 31 20:06:44.293573 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:44.293520    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:06:47.046575 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:47.046464    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:06:49.583047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:49.583008    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"]
Oct 31 20:06:49.583557 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:49.583278    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" containerID="cri-o://4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71" gracePeriod=30
Oct 31 20:06:53.394214 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:53.394159    5958 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:06:53.394932 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:53.394247    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:06:54.877046 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.877011    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:06:54.962922 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.962886    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.962938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963016    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963045    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963327 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963217    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963327 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963281    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963444 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963315    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.963444 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963352    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963444 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963384    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.963444 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963413    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert\") pod \"a2140f87-1dde-4114-924f-7eb27f801109\" (UID: \"a2140f87-1dde-4114-924f-7eb27f801109\") "
Oct 31 20:06:54.964182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.963767    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.964182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.964095    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.964182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.964118    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.964182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.964140    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.964182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.964147    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.964485 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.964300    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:54.965644 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.965603    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "kube-api-access-dvk7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:54.965791 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.965675    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:54.965791 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.965719    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:54.966133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:54.966111    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy" pod "a2140f87-1dde-4114-924f-7eb27f801109" (UID: "a2140f87-1dde-4114-924f-7eb27f801109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:55.065813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065776    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvk7k\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-kube-api-access-dvk7k\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.065813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065810    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.065813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065821    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2140f87-1dde-4114-924f-7eb27f801109-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.066164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065830    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.066164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065840    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2140f87-1dde-4114-924f-7eb27f801109-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.066164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065849    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2140f87-1dde-4114-924f-7eb27f801109-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.066164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.065857    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2140f87-1dde-4114-924f-7eb27f801109-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.520379 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.520342    5958 generic.go:356] "Generic (PLEG): container finished" podID="a2140f87-1dde-4114-924f-7eb27f801109" containerID="4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71" exitCode=0
Oct 31 20:06:55.520619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.520406    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"
Oct 31 20:06:55.520619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.520451    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" event={"ID":"a2140f87-1dde-4114-924f-7eb27f801109","Type":"ContainerDied","Data":"4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71"}
Oct 31 20:06:55.520619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.520517    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc" event={"ID":"a2140f87-1dde-4114-924f-7eb27f801109","Type":"ContainerDied","Data":"1569f8051d202134b89b7cfd7294828e6323a5be3a2ff6e474689524708eac53"}
Oct 31 20:06:55.520619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.520560    5958 scope.go:122] "RemoveContainer" containerID="4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71"
Oct 31 20:06:55.531499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.531479    5958 scope.go:122] "RemoveContainer" containerID="4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71"
Oct 31 20:06:55.531836 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:06:55.531815    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71\": container with ID starting with 4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71 not found: ID does not exist" containerID="4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71"
Oct 31 20:06:55.531886 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.531845    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71"} err="failed to get container status \"4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71\": rpc error: code = NotFound desc = could not find container \"4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71\": container with ID starting with 4dede92b344130ff0823c88a443d2f3b476dbab37141a72fe095a18e553b7c71 not found: ID does not exist"
Oct 31 20:06:55.551144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.551109    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"]
Oct 31 20:06:55.553980 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:55.553961    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-9sjnc"]
Oct 31 20:06:56.663976 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:06:56.663941    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2140f87-1dde-4114-924f-7eb27f801109" path="/var/lib/kubelet/pods/a2140f87-1dde-4114-924f-7eb27f801109/volumes"
Oct 31 20:07:01.691922 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:01.691887    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"]
Oct 31 20:07:01.692311 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:01.692081    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" podUID="5730e849-2fe7-41a5-894f-3def0ee02cb8" containerName="maas-api" containerID="cri-o://98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571" gracePeriod=30
Oct 31 20:07:01.824730 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:01.824691    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"]
Oct 31 20:07:01.824977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:01.824951    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs" podUID="5a2738d1-4587-4dfd-8ea4-2542d0d9b842" containerName="payload-processing" containerID="cri-o://52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045" gracePeriod=30
Oct 31 20:07:01.834904 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:01.834424    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"]
Oct 31 20:07:01.834904 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:01.834693    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2" podUID="79d8b407-edfa-4b0b-925d-ecaf2d014038" containerName="payload-pre-processing" containerID="cri-o://f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893" gracePeriod=30
Oct 31 20:07:02.013333 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.012825    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:07:02.138300 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.138255    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:07:02.153986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.151194    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls\") pod \"5730e849-2fe7-41a5-894f-3def0ee02cb8\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") "
Oct 31 20:07:02.153986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.151257    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca\") pod \"5730e849-2fe7-41a5-894f-3def0ee02cb8\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") "
Oct 31 20:07:02.153986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.151298    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5\" (UniqueName: \"kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5\") pod \"5730e849-2fe7-41a5-894f-3def0ee02cb8\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") "
Oct 31 20:07:02.153986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.151364    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs\") pod \"5730e849-2fe7-41a5-894f-3def0ee02cb8\" (UID: \"5730e849-2fe7-41a5-894f-3def0ee02cb8\") "
Oct 31 20:07:02.156772 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.156676    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca" pod "5730e849-2fe7-41a5-894f-3def0ee02cb8" (UID: "5730e849-2fe7-41a5-894f-3def0ee02cb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:02.157578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.157520    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5" pod "5730e849-2fe7-41a5-894f-3def0ee02cb8" (UID: "5730e849-2fe7-41a5-894f-3def0ee02cb8"). InnerVolumeSpecName "kube-api-access-4flb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:02.162053 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.162012    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs" pod "5730e849-2fe7-41a5-894f-3def0ee02cb8" (UID: "5730e849-2fe7-41a5-894f-3def0ee02cb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:02.162943 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.162900    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls" pod "5730e849-2fe7-41a5-894f-3def0ee02cb8" (UID: "5730e849-2fe7-41a5-894f-3def0ee02cb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:02.209227 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.208832    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:07:02.253296 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.253261    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume\") pod \"79d8b407-edfa-4b0b-925d-ecaf2d014038\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") "
Oct 31 20:07:02.253527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.253396    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume\") pod \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") "
Oct 31 20:07:02.253527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.253468    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24\" (UniqueName: \"kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24\") pod \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\" (UID: \"5a2738d1-4587-4dfd-8ea4-2542d0d9b842\") "
Oct 31 20:07:02.253527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.253522    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc\" (UniqueName: \"kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc\") pod \"79d8b407-edfa-4b0b-925d-ecaf2d014038\" (UID: \"79d8b407-edfa-4b0b-925d-ecaf2d014038\") "
Oct 31 20:07:02.253854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.253810    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume" pod "79d8b407-edfa-4b0b-925d-ecaf2d014038" (UID: "79d8b407-edfa-4b0b-925d-ecaf2d014038"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:02.253854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.253818    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume" pod "5a2738d1-4587-4dfd-8ea4-2542d0d9b842" (UID: "5a2738d1-4587-4dfd-8ea4-2542d0d9b842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:02.254087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.254066    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.254155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.254092    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8b407-edfa-4b0b-925d-ecaf2d014038-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.254155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.254108    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.254155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.254122    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730e849-2fe7-41a5-894f-3def0ee02cb8-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.254155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.254136    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4flb5\" (UniqueName: \"kubernetes.io/projected/5730e849-2fe7-41a5-894f-3def0ee02cb8-kube-api-access-4flb5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.254155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.254152    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5730e849-2fe7-41a5-894f-3def0ee02cb8-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.256603 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.256532    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc" pod "79d8b407-edfa-4b0b-925d-ecaf2d014038" (UID: "79d8b407-edfa-4b0b-925d-ecaf2d014038"). InnerVolumeSpecName "kube-api-access-mj2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:02.256740 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.256672    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24" pod "5a2738d1-4587-4dfd-8ea4-2542d0d9b842" (UID: "5a2738d1-4587-4dfd-8ea4-2542d0d9b842"). InnerVolumeSpecName "kube-api-access-w8f24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:02.354982 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.354942    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8f24\" (UniqueName: \"kubernetes.io/projected/5a2738d1-4587-4dfd-8ea4-2542d0d9b842-kube-api-access-w8f24\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.354982 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.354975    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2nc\" (UniqueName: \"kubernetes.io/projected/79d8b407-edfa-4b0b-925d-ecaf2d014038-kube-api-access-mj2nc\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:02.565566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.565458    5958 generic.go:356] "Generic (PLEG): container finished" podID="5730e849-2fe7-41a5-894f-3def0ee02cb8" containerID="98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571" exitCode=0
Oct 31 20:07:02.565566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.565501    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"
Oct 31 20:07:02.565815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.565582    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" event={"ID":"5730e849-2fe7-41a5-894f-3def0ee02cb8","Type":"ContainerDied","Data":"98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571"}
Oct 31 20:07:02.565815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.565634    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" event={"ID":"5730e849-2fe7-41a5-894f-3def0ee02cb8","Type":"ContainerDied","Data":"c34f1b1e1c24006e2bbd762b4baf68e0044bdb917db79a939fc563c4cf4bdb66"}
Oct 31 20:07:02.565815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.565666    5958 scope.go:122] "RemoveContainer" containerID="98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571"
Oct 31 20:07:02.567234 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.567207    5958 generic.go:356] "Generic (PLEG): container finished" podID="79d8b407-edfa-4b0b-925d-ecaf2d014038" containerID="f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893" exitCode=0
Oct 31 20:07:02.567372 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.567260    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"
Oct 31 20:07:02.567372 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.567286    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2" event={"ID":"79d8b407-edfa-4b0b-925d-ecaf2d014038","Type":"ContainerDied","Data":"f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893"}
Oct 31 20:07:02.567372 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.567330    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2" event={"ID":"79d8b407-edfa-4b0b-925d-ecaf2d014038","Type":"ContainerDied","Data":"f378bcaed1458b063a005a22b7a4d75da17ca173a6c6058fa65217337c942474"}
Oct 31 20:07:02.568774 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.568752    5958 generic.go:356] "Generic (PLEG): container finished" podID="5a2738d1-4587-4dfd-8ea4-2542d0d9b842" containerID="52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045" exitCode=0
Oct 31 20:07:02.568889 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.568785    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"
Oct 31 20:07:02.568889 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.568826    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs" event={"ID":"5a2738d1-4587-4dfd-8ea4-2542d0d9b842","Type":"ContainerDied","Data":"52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045"}
Oct 31 20:07:02.568889 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.568876    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs" event={"ID":"5a2738d1-4587-4dfd-8ea4-2542d0d9b842","Type":"ContainerDied","Data":"1b5dd8d204da76b28f9410b554571d2bc69da79ceb45a3a08b6bf3285486a878"}
Oct 31 20:07:02.576523 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.576502    5958 scope.go:122] "RemoveContainer" containerID="98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571"
Oct 31 20:07:02.576921 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:02.576898    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571\": container with ID starting with 98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571 not found: ID does not exist" containerID="98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571"
Oct 31 20:07:02.577011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.576933    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571"} err="failed to get container status \"98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571\": rpc error: code = NotFound desc = could not find container \"98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571\": container with ID starting with 98cd3861a3b7309d511fa3e2cd320ebb67ceebf880859b6c42d37f4188c50571 not found: ID does not exist"
Oct 31 20:07:02.577011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.576958    5958 scope.go:122] "RemoveContainer" containerID="f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893"
Oct 31 20:07:02.586757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.586739    5958 scope.go:122] "RemoveContainer" containerID="f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893"
Oct 31 20:07:02.587040 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:02.587018    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893\": container with ID starting with f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893 not found: ID does not exist" containerID="f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893"
Oct 31 20:07:02.587118 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.587053    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893"} err="failed to get container status \"f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893\": rpc error: code = NotFound desc = could not find container \"f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893\": container with ID starting with f8a6e010d21b55ac0b6a361a0e089e1b94eb83555b842ab65629844d8cd7a893 not found: ID does not exist"
Oct 31 20:07:02.587118 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.587075    5958 scope.go:122] "RemoveContainer" containerID="52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045"
Oct 31 20:07:02.596885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.596859    5958 scope.go:122] "RemoveContainer" containerID="52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045"
Oct 31 20:07:02.597196 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:02.597176    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045\": container with ID starting with 52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045 not found: ID does not exist" containerID="52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045"
Oct 31 20:07:02.597288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.597205    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045"} err="failed to get container status \"52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045\": rpc error: code = NotFound desc = could not find container \"52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045\": container with ID starting with 52f43cbe17fda8690f3ebd092dbc2ca2c002642eaaea334901d9c72060472045 not found: ID does not exist"
Oct 31 20:07:02.602088 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.602063    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"]
Oct 31 20:07:02.608346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.608323    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7fe845-7c7f967f6b-dqxhs"]
Oct 31 20:07:02.623772 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.623743    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"]
Oct 31 20:07:02.628368 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.628343    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7fe845-754c6876d7-4j9n2"]
Oct 31 20:07:02.642974 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.642940    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"]
Oct 31 20:07:02.646467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.646441    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82"]
Oct 31 20:07:02.663411 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.663382    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5730e849-2fe7-41a5-894f-3def0ee02cb8" path="/var/lib/kubelet/pods/5730e849-2fe7-41a5-894f-3def0ee02cb8/volumes"
Oct 31 20:07:02.663830 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.663815    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2738d1-4587-4dfd-8ea4-2542d0d9b842" path="/var/lib/kubelet/pods/5a2738d1-4587-4dfd-8ea4-2542d0d9b842/volumes"
Oct 31 20:07:02.664168 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:02.664155    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d8b407-edfa-4b0b-925d-ecaf2d014038" path="/var/lib/kubelet/pods/79d8b407-edfa-4b0b-925d-ecaf2d014038/volumes"
Oct 31 20:07:04.935763 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:04.935693    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7fe845-7ccb7c5b68-6md82" podUID="5730e849-2fe7-41a5-894f-3def0ee02cb8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.239:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:07:06.164719 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:06.164679    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:07:15.121452 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:15.121412    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:07:17.046503 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:17.046455    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:07:22.145490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:22.145449    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"]
Oct 31 20:07:22.168165 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:22.168130    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"]
Oct 31 20:07:23.318737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.318705    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"]
Oct 31 20:07:23.319201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.318914    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" podUID="128d8a3d-d166-482f-9e12-00ebf955e68b" containerName="maas-api" containerID="cri-o://3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030" gracePeriod=30
Oct 31 20:07:23.600773 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.600746    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 20:07:23.616588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.616514    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs\") pod \"128d8a3d-d166-482f-9e12-00ebf955e68b\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") "
Oct 31 20:07:23.616588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.616585    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\") pod \"128d8a3d-d166-482f-9e12-00ebf955e68b\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") "
Oct 31 20:07:23.616868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.616633    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca\") pod \"128d8a3d-d166-482f-9e12-00ebf955e68b\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") "
Oct 31 20:07:23.616868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.616676    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7\" (UniqueName: \"kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7\") pod \"128d8a3d-d166-482f-9e12-00ebf955e68b\" (UID: \"128d8a3d-d166-482f-9e12-00ebf955e68b\") "
Oct 31 20:07:23.617020 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.616995    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca" pod "128d8a3d-d166-482f-9e12-00ebf955e68b" (UID: "128d8a3d-d166-482f-9e12-00ebf955e68b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:23.617229 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.617209    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/128d8a3d-d166-482f-9e12-00ebf955e68b-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:23.619295 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.619265    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7" pod "128d8a3d-d166-482f-9e12-00ebf955e68b" (UID: "128d8a3d-d166-482f-9e12-00ebf955e68b"). InnerVolumeSpecName "kube-api-access-wslf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:23.619854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.619823    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs" pod "128d8a3d-d166-482f-9e12-00ebf955e68b" (UID: "128d8a3d-d166-482f-9e12-00ebf955e68b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:23.620131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.620111    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls" pod "128d8a3d-d166-482f-9e12-00ebf955e68b" (UID: "128d8a3d-d166-482f-9e12-00ebf955e68b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:23.718904 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.718860    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:23.718904 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.718901    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/128d8a3d-d166-482f-9e12-00ebf955e68b-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:23.719133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.718919    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wslf7\" (UniqueName: \"kubernetes.io/projected/128d8a3d-d166-482f-9e12-00ebf955e68b-kube-api-access-wslf7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:23.725482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.725453    5958 generic.go:356] "Generic (PLEG): container finished" podID="128d8a3d-d166-482f-9e12-00ebf955e68b" containerID="3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030" exitCode=0
Oct 31 20:07:23.725688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.725514    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"
Oct 31 20:07:23.725688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.725571    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" event={"ID":"128d8a3d-d166-482f-9e12-00ebf955e68b","Type":"ContainerDied","Data":"3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030"}
Oct 31 20:07:23.725688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.725641    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb" event={"ID":"128d8a3d-d166-482f-9e12-00ebf955e68b","Type":"ContainerDied","Data":"68d0c4945404770711ff3745e038c33bf9d9959b974c909555a9d12ea405f603"}
Oct 31 20:07:23.725688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.725673    5958 scope.go:122] "RemoveContainer" containerID="3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030"
Oct 31 20:07:23.737089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.737058    5958 scope.go:122] "RemoveContainer" containerID="3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030"
Oct 31 20:07:23.737422 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:23.737397    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030\": container with ID starting with 3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030 not found: ID does not exist" containerID="3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030"
Oct 31 20:07:23.737474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.737434    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030"} err="failed to get container status \"3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030\": rpc error: code = NotFound desc = could not find container \"3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030\": container with ID starting with 3c1248bfd2b05f6ee1aef514bfa73bd6fba96c799821c66eb69656bf04d75030 not found: ID does not exist"
Oct 31 20:07:23.756086 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.756059    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"]
Oct 31 20:07:23.760335 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:23.760311    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cd9729-6db597b959-8d6mb"]
Oct 31 20:07:24.664074 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:24.664037    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128d8a3d-d166-482f-9e12-00ebf955e68b" path="/var/lib/kubelet/pods/128d8a3d-d166-482f-9e12-00ebf955e68b/volumes"
Oct 31 20:07:25.849100 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:25.849059    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"]
Oct 31 20:07:25.864437 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:25.864403    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"]
Oct 31 20:07:27.146594 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:27.146515    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerName="payload-processing" containerID="cri-o://7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590" gracePeriod=25
Oct 31 20:07:27.169048 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:27.169004    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerName="payload-pre-processing" containerID="cri-o://eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f" gracePeriod=25
Oct 31 20:07:28.051578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:28.051528    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:07:29.541932 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.541891    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"]
Oct 31 20:07:29.542834 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.542793    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" podUID="dc60ec47-7d8b-4643-8b18-5288ac867803" containerName="maas-api" containerID="cri-o://8273a3bcef2799ea1dbd689a414028e9e70ccb80ac7f335f46aa69bbe331884f" gracePeriod=30
Oct 31 20:07:29.779001 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.778932    5958 generic.go:356] "Generic (PLEG): container finished" podID="dc60ec47-7d8b-4643-8b18-5288ac867803" containerID="8273a3bcef2799ea1dbd689a414028e9e70ccb80ac7f335f46aa69bbe331884f" exitCode=0
Oct 31 20:07:29.779357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.779330    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" event={"ID":"dc60ec47-7d8b-4643-8b18-5288ac867803","Type":"ContainerDied","Data":"8273a3bcef2799ea1dbd689a414028e9e70ccb80ac7f335f46aa69bbe331884f"}
Oct 31 20:07:29.840266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.840239    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 20:07:29.903344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.903305    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\") pod \"dc60ec47-7d8b-4643-8b18-5288ac867803\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") "
Oct 31 20:07:29.903344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.903406    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs\") pod \"dc60ec47-7d8b-4643-8b18-5288ac867803\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") "
Oct 31 20:07:29.904814 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.903588    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j\" (UniqueName: \"kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j\") pod \"dc60ec47-7d8b-4643-8b18-5288ac867803\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") "
Oct 31 20:07:29.904814 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.903633    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca\") pod \"dc60ec47-7d8b-4643-8b18-5288ac867803\" (UID: \"dc60ec47-7d8b-4643-8b18-5288ac867803\") "
Oct 31 20:07:29.905480 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.905442    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca" pod "dc60ec47-7d8b-4643-8b18-5288ac867803" (UID: "dc60ec47-7d8b-4643-8b18-5288ac867803"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:29.906856 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.906822    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j" pod "dc60ec47-7d8b-4643-8b18-5288ac867803" (UID: "dc60ec47-7d8b-4643-8b18-5288ac867803"). InnerVolumeSpecName "kube-api-access-7v42j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:29.906993 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.906909    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs" pod "dc60ec47-7d8b-4643-8b18-5288ac867803" (UID: "dc60ec47-7d8b-4643-8b18-5288ac867803"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:29.907986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:29.907964    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls" pod "dc60ec47-7d8b-4643-8b18-5288ac867803" (UID: "dc60ec47-7d8b-4643-8b18-5288ac867803"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:30.010522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.010468    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v42j\" (UniqueName: \"kubernetes.io/projected/dc60ec47-7d8b-4643-8b18-5288ac867803-kube-api-access-7v42j\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:30.010522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.010511    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc60ec47-7d8b-4643-8b18-5288ac867803-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:30.010522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.010528    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:30.010881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.010558    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc60ec47-7d8b-4643-8b18-5288ac867803-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:30.786957 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.786920    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"
Oct 31 20:07:30.787417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.786953    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2" event={"ID":"dc60ec47-7d8b-4643-8b18-5288ac867803","Type":"ContainerDied","Data":"98755f95f2161ca043b6a329f121cf27d4171a073c5fc6c6a293159ec13f305d"}
Oct 31 20:07:30.787417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.787021    5958 scope.go:122] "RemoveContainer" containerID="8273a3bcef2799ea1dbd689a414028e9e70ccb80ac7f335f46aa69bbe331884f"
Oct 31 20:07:30.816212 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.816174    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"]
Oct 31 20:07:30.818525 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.818496    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c55cf1-5cd44d96cd-xxcs2"]
Oct 31 20:07:30.849464 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.849419    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerName="payload-processing" containerID="cri-o://abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9" gracePeriod=25
Oct 31 20:07:30.865705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:30.865642    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerName="payload-pre-processing" containerID="cri-o://333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4" gracePeriod=25
Oct 31 20:07:31.871346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:31.871299    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:31.871346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:31.871335    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:32.663634 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:32.663599    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc60ec47-7d8b-4643-8b18-5288ac867803" path="/var/lib/kubelet/pods/dc60ec47-7d8b-4643-8b18-5288ac867803/volumes"
Oct 31 20:07:39.448796 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:39.448751    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:40.442071 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:40.442027    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:07:40.458724 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:40.458683    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:41.870478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:41.870435    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:41.870918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:41.870441    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:43.166344 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:43.166289    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"]
Oct 31 20:07:43.197340 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:43.197296    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"]
Oct 31 20:07:47.047146 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:47.047105    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:07:47.173480 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.173449    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"]
Oct 31 20:07:47.173744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.173703    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" podUID="3d88c070-0722-481d-9056-1f2aa89a02a3" containerName="maas-api" containerID="cri-o://e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085" gracePeriod=30
Oct 31 20:07:47.510048 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.509937    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 20:07:47.514677 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.514582    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 20:07:47.576474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.576442    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 20:07:47.623763 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.623726    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs\") pod \"3d88c070-0722-481d-9056-1f2aa89a02a3\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") "
Oct 31 20:07:47.623972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.623794    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\") pod \"3d88c070-0722-481d-9056-1f2aa89a02a3\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") "
Oct 31 20:07:47.623972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.623901    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84\" (UniqueName: \"kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84\") pod \"4ae17531-6e12-4803-9977-cb2e9a349df5\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") "
Oct 31 20:07:47.623972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.623952    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume\") pod \"072664a6-4f1d-4318-b9e4-b31190d80f64\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") "
Oct 31 20:07:47.624147 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.624028    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca\") pod \"3d88c070-0722-481d-9056-1f2aa89a02a3\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") "
Oct 31 20:07:47.624147 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.624077    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume\") pod \"4ae17531-6e12-4803-9977-cb2e9a349df5\" (UID: \"4ae17531-6e12-4803-9977-cb2e9a349df5\") "
Oct 31 20:07:47.624147 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.624104    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf\" (UniqueName: \"kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf\") pod \"3d88c070-0722-481d-9056-1f2aa89a02a3\" (UID: \"3d88c070-0722-481d-9056-1f2aa89a02a3\") "
Oct 31 20:07:47.624304 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.624150    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw\" (UniqueName: \"kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw\") pod \"072664a6-4f1d-4318-b9e4-b31190d80f64\" (UID: \"072664a6-4f1d-4318-b9e4-b31190d80f64\") "
Oct 31 20:07:47.624871 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.624840    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume" pod "072664a6-4f1d-4318-b9e4-b31190d80f64" (UID: "072664a6-4f1d-4318-b9e4-b31190d80f64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:47.625177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.625144    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume" pod "4ae17531-6e12-4803-9977-cb2e9a349df5" (UID: "4ae17531-6e12-4803-9977-cb2e9a349df5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:47.625615 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.625577    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca" pod "3d88c070-0722-481d-9056-1f2aa89a02a3" (UID: "3d88c070-0722-481d-9056-1f2aa89a02a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:47.627388 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.627349    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs" pod "3d88c070-0722-481d-9056-1f2aa89a02a3" (UID: "3d88c070-0722-481d-9056-1f2aa89a02a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:47.627531 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.627368    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls" pod "3d88c070-0722-481d-9056-1f2aa89a02a3" (UID: "3d88c070-0722-481d-9056-1f2aa89a02a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:47.628108 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.628086    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84" pod "4ae17531-6e12-4803-9977-cb2e9a349df5" (UID: "4ae17531-6e12-4803-9977-cb2e9a349df5"). InnerVolumeSpecName "kube-api-access-6nm84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:47.629883 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.629855    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw" pod "072664a6-4f1d-4318-b9e4-b31190d80f64" (UID: "072664a6-4f1d-4318-b9e4-b31190d80f64"). InnerVolumeSpecName "kube-api-access-gfddw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:47.631210 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.631181    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf" pod "3d88c070-0722-481d-9056-1f2aa89a02a3" (UID: "3d88c070-0722-481d-9056-1f2aa89a02a3"). InnerVolumeSpecName "kube-api-access-5bjwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:47.731592 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.730425    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfddw\" (UniqueName: \"kubernetes.io/projected/072664a6-4f1d-4318-b9e4-b31190d80f64-kube-api-access-gfddw\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.731881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.731857    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.731990 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.731977    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d88c070-0722-481d-9056-1f2aa89a02a3-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.732085 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.732074    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nm84\" (UniqueName: \"kubernetes.io/projected/4ae17531-6e12-4803-9977-cb2e9a349df5-kube-api-access-6nm84\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.732167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.732156    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/072664a6-4f1d-4318-b9e4-b31190d80f64-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.732269 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.732256    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d88c070-0722-481d-9056-1f2aa89a02a3-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.732447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.732356    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae17531-6e12-4803-9977-cb2e9a349df5-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.732447 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.732377    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjwf\" (UniqueName: \"kubernetes.io/projected/3d88c070-0722-481d-9056-1f2aa89a02a3-kube-api-access-5bjwf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:47.904261 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.904169    5958 generic.go:356] "Generic (PLEG): container finished" podID="3d88c070-0722-481d-9056-1f2aa89a02a3" containerID="e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085" exitCode=0
Oct 31 20:07:47.904261 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.904216    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"
Oct 31 20:07:47.904519 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.904255    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" event={"ID":"3d88c070-0722-481d-9056-1f2aa89a02a3","Type":"ContainerDied","Data":"e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085"}
Oct 31 20:07:47.904519 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.904312    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7" event={"ID":"3d88c070-0722-481d-9056-1f2aa89a02a3","Type":"ContainerDied","Data":"4cf34c6f3868f0693b09b8e15a58cc9129a84f27d9d3ebff58ab54670e6157d5"}
Oct 31 20:07:47.904519 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.904342    5958 scope.go:122] "RemoveContainer" containerID="e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085"
Oct 31 20:07:47.905914 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.905884    5958 generic.go:356] "Generic (PLEG): container finished" podID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerID="eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f" exitCode=0
Oct 31 20:07:47.906034 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.905947    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"
Oct 31 20:07:47.906034 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.905974    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" event={"ID":"4ae17531-6e12-4803-9977-cb2e9a349df5","Type":"ContainerDied","Data":"eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f"}
Oct 31 20:07:47.906167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.906042    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6" event={"ID":"4ae17531-6e12-4803-9977-cb2e9a349df5","Type":"ContainerDied","Data":"55402883db2f733fb536ee3f43fde10269404cf629375f1565421baa61e3b412"}
Oct 31 20:07:47.907703 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.907678    5958 generic.go:356] "Generic (PLEG): container finished" podID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerID="7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590" exitCode=0
Oct 31 20:07:47.907833 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.907722    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"
Oct 31 20:07:47.908058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.907722    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" event={"ID":"072664a6-4f1d-4318-b9e4-b31190d80f64","Type":"ContainerDied","Data":"7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590"}
Oct 31 20:07:47.908058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.908024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w" event={"ID":"072664a6-4f1d-4318-b9e4-b31190d80f64","Type":"ContainerDied","Data":"28193b7917516a854e186ab1d69394ed55038ed73466bb2ed6cb9a5889afc069"}
Oct 31 20:07:47.916622 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.916040    5958 scope.go:122] "RemoveContainer" containerID="e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085"
Oct 31 20:07:47.919015 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:47.917406    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085\": container with ID starting with e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085 not found: ID does not exist" containerID="e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085"
Oct 31 20:07:47.919015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.917462    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085"} err="failed to get container status \"e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085\": rpc error: code = NotFound desc = could not find container \"e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085\": container with ID starting with e55eaef938c63ef6a45009409d5ba7657d6e1a426811551e1288e6f761be2085 not found: ID does not exist"
Oct 31 20:07:47.919015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.917491    5958 scope.go:122] "RemoveContainer" containerID="eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f"
Oct 31 20:07:47.938084 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.938062    5958 scope.go:122] "RemoveContainer" containerID="eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f"
Oct 31 20:07:47.938425 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:47.938401    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f\": container with ID starting with eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f not found: ID does not exist" containerID="eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f"
Oct 31 20:07:47.938484 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.938436    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f"} err="failed to get container status \"eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f\": rpc error: code = NotFound desc = could not find container \"eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f\": container with ID starting with eec75dc309af85c13dfaa67a1794f04333d219633b3cd303f5f8ca0a5f79610f not found: ID does not exist"
Oct 31 20:07:47.938484 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.938453    5958 scope.go:122] "RemoveContainer" containerID="7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590"
Oct 31 20:07:47.943022 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.942998    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"]
Oct 31 20:07:47.950921 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.950896    5958 scope.go:122] "RemoveContainer" containerID="7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590"
Oct 31 20:07:47.950921 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.950913    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cd9729-77b49dfbbc-m7jw6"]
Oct 31 20:07:47.951249 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:47.951230    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590\": container with ID starting with 7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590 not found: ID does not exist" containerID="7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590"
Oct 31 20:07:47.951291 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.951259    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590"} err="failed to get container status \"7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590\": rpc error: code = NotFound desc = could not find container \"7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590\": container with ID starting with 7403f6ca697a9c3e003f93c6f87603320e1fff4ff5b8c1096b5f0462dcc4e590 not found: ID does not exist"
Oct 31 20:07:47.964376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.964349    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"]
Oct 31 20:07:47.971799 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.971771    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cd9729-774576d6f5-g8n6w"]
Oct 31 20:07:47.986614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.986577    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"]
Oct 31 20:07:47.990570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:47.990524    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-435728-6d59644df5-wvzf7"]
Oct 31 20:07:48.166882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:48.166840    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" podUID="efbe721a-665b-4c89-b288-0d295946bba3" containerName="payload-processing" containerID="cri-o://d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f" gracePeriod=25
Oct 31 20:07:48.197705 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:48.197667    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerName="payload-pre-processing" containerID="cri-o://af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c" gracePeriod=25
Oct 31 20:07:48.664646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:48.664617    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" path="/var/lib/kubelet/pods/072664a6-4f1d-4318-b9e4-b31190d80f64/volumes"
Oct 31 20:07:48.665073 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:48.665057    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d88c070-0722-481d-9056-1f2aa89a02a3" path="/var/lib/kubelet/pods/3d88c070-0722-481d-9056-1f2aa89a02a3/volumes"
Oct 31 20:07:48.665466 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:48.665452    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" path="/var/lib/kubelet/pods/4ae17531-6e12-4803-9977-cb2e9a349df5/volumes"
Oct 31 20:07:49.448653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:49.448609    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:49.563936 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:49.563899    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"]
Oct 31 20:07:49.635486 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:49.635455    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"]
Oct 31 20:07:50.458764 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:50.458724    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:51.132173 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.132145    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 20:07:51.174455 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.174425    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn\" (UniqueName: \"kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn\") pod \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") "
Oct 31 20:07:51.174653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.174489    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume\") pod \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\" (UID: \"b37734fa-f33f-4d92-b6ce-0a81b16291d0\") "
Oct 31 20:07:51.174929 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.174894    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume" pod "b37734fa-f33f-4d92-b6ce-0a81b16291d0" (UID: "b37734fa-f33f-4d92-b6ce-0a81b16291d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:51.176734 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.176711    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn" pod "b37734fa-f33f-4d92-b6ce-0a81b16291d0" (UID: "b37734fa-f33f-4d92-b6ce-0a81b16291d0"). InnerVolumeSpecName "kube-api-access-nrpbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:51.181993 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.181965    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 20:07:51.231603 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.231565    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"]
Oct 31 20:07:51.231854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.231810    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" podUID="420f78c4-3d9b-43b3-8115-f00fa70f956e" containerName="maas-api" containerID="cri-o://4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f" gracePeriod=30
Oct 31 20:07:51.276167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.276135    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume\") pod \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") "
Oct 31 20:07:51.276355 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.276183    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf\" (UniqueName: \"kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf\") pod \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\" (UID: \"619b2c9c-f018-491e-b7e8-dae9abd7ccdc\") "
Oct 31 20:07:51.276534 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.276513    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpbn\" (UniqueName: \"kubernetes.io/projected/b37734fa-f33f-4d92-b6ce-0a81b16291d0-kube-api-access-nrpbn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.276615 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.276538    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b37734fa-f33f-4d92-b6ce-0a81b16291d0-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.276658 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.276606    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume" pod "619b2c9c-f018-491e-b7e8-dae9abd7ccdc" (UID: "619b2c9c-f018-491e-b7e8-dae9abd7ccdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:51.278315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.278290    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf" pod "619b2c9c-f018-491e-b7e8-dae9abd7ccdc" (UID: "619b2c9c-f018-491e-b7e8-dae9abd7ccdc"). InnerVolumeSpecName "kube-api-access-b9brf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:51.377361 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.377320    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.377361 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.377359    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9brf\" (UniqueName: \"kubernetes.io/projected/619b2c9c-f018-491e-b7e8-dae9abd7ccdc-kube-api-access-b9brf\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.501931 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.501904    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 20:07:51.580476 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.580442    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\") pod \"420f78c4-3d9b-43b3-8115-f00fa70f956e\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") "
Oct 31 20:07:51.580476 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.580488    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs\") pod \"420f78c4-3d9b-43b3-8115-f00fa70f956e\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") "
Oct 31 20:07:51.580744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.580526    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca\") pod \"420f78c4-3d9b-43b3-8115-f00fa70f956e\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") "
Oct 31 20:07:51.580744 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.580616    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b\" (UniqueName: \"kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b\") pod \"420f78c4-3d9b-43b3-8115-f00fa70f956e\" (UID: \"420f78c4-3d9b-43b3-8115-f00fa70f956e\") "
Oct 31 20:07:51.581038 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.581010    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca" pod "420f78c4-3d9b-43b3-8115-f00fa70f956e" (UID: "420f78c4-3d9b-43b3-8115-f00fa70f956e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:51.582800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.582772    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs" pod "420f78c4-3d9b-43b3-8115-f00fa70f956e" (UID: "420f78c4-3d9b-43b3-8115-f00fa70f956e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:51.582918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.582823    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b" pod "420f78c4-3d9b-43b3-8115-f00fa70f956e" (UID: "420f78c4-3d9b-43b3-8115-f00fa70f956e"). InnerVolumeSpecName "kube-api-access-2c89b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:51.582918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.582827    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls" pod "420f78c4-3d9b-43b3-8115-f00fa70f956e" (UID: "420f78c4-3d9b-43b3-8115-f00fa70f956e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:51.681975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.681870    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c89b\" (UniqueName: \"kubernetes.io/projected/420f78c4-3d9b-43b3-8115-f00fa70f956e-kube-api-access-2c89b\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.681975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.681911    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.681975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.681926    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/420f78c4-3d9b-43b3-8115-f00fa70f956e-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.681975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.681941    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/420f78c4-3d9b-43b3-8115-f00fa70f956e-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:51.940396 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.940306    5958 generic.go:356] "Generic (PLEG): container finished" podID="420f78c4-3d9b-43b3-8115-f00fa70f956e" containerID="4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f" exitCode=0
Oct 31 20:07:51.940396 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.940350    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"
Oct 31 20:07:51.940653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.940394    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" event={"ID":"420f78c4-3d9b-43b3-8115-f00fa70f956e","Type":"ContainerDied","Data":"4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f"}
Oct 31 20:07:51.940653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.940444    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z" event={"ID":"420f78c4-3d9b-43b3-8115-f00fa70f956e","Type":"ContainerDied","Data":"6cbba8ef52052d91295d69f62f6bdfbe4ff4201d3b053fee65266c180035d2f7"}
Oct 31 20:07:51.940653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.940468    5958 scope.go:122] "RemoveContainer" containerID="4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f"
Oct 31 20:07:51.942022 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.941996    5958 generic.go:356] "Generic (PLEG): container finished" podID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerID="abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9" exitCode=0
Oct 31 20:07:51.942165 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.942045    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"
Oct 31 20:07:51.942165 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.942085    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" event={"ID":"b37734fa-f33f-4d92-b6ce-0a81b16291d0","Type":"ContainerDied","Data":"abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9"}
Oct 31 20:07:51.942165 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.942136    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg" event={"ID":"b37734fa-f33f-4d92-b6ce-0a81b16291d0","Type":"ContainerDied","Data":"e6101e39756e87f0c6817f24dd346a111618f669db1cfcdeedf066985cd9aa05"}
Oct 31 20:07:51.943832 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.943805    5958 generic.go:356] "Generic (PLEG): container finished" podID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerID="333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4" exitCode=0
Oct 31 20:07:51.944014 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.943842    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" event={"ID":"619b2c9c-f018-491e-b7e8-dae9abd7ccdc","Type":"ContainerDied","Data":"333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4"}
Oct 31 20:07:51.944014 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.943879    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2" event={"ID":"619b2c9c-f018-491e-b7e8-dae9abd7ccdc","Type":"ContainerDied","Data":"959e5905e9ea406fb15a7382b76e8d9e3b0315283913f62634d7fc33823e620b"}
Oct 31 20:07:51.944014 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.943907    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"
Oct 31 20:07:51.953604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.953583    5958 scope.go:122] "RemoveContainer" containerID="4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f"
Oct 31 20:07:51.953886 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:51.953866    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f\": container with ID starting with 4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f not found: ID does not exist" containerID="4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f"
Oct 31 20:07:51.953988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.953894    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f"} err="failed to get container status \"4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f\": rpc error: code = NotFound desc = could not find container \"4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f\": container with ID starting with 4bc7a037200cfe85ca3d78425468cc22bfde79a3c1986720524fcc3b5bb4504f not found: ID does not exist"
Oct 31 20:07:51.953988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.953911    5958 scope.go:122] "RemoveContainer" containerID="abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9"
Oct 31 20:07:51.963475 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.963456    5958 scope.go:122] "RemoveContainer" containerID="abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9"
Oct 31 20:07:51.964133 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:51.963727    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9\": container with ID starting with abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9 not found: ID does not exist" containerID="abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9"
Oct 31 20:07:51.964133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.963747    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9"} err="failed to get container status \"abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9\": rpc error: code = NotFound desc = could not find container \"abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9\": container with ID starting with abbc9c7c0708f558a4aff95ffcebdf72d4af0e095e395bd24ef7cc8f0b6c90e9 not found: ID does not exist"
Oct 31 20:07:51.964133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.963762    5958 scope.go:122] "RemoveContainer" containerID="333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4"
Oct 31 20:07:51.973855 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.973572    5958 scope.go:122] "RemoveContainer" containerID="333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4"
Oct 31 20:07:51.973973 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:51.973929    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4\": container with ID starting with 333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4 not found: ID does not exist" containerID="333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4"
Oct 31 20:07:51.974043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.973966    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4"} err="failed to get container status \"333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4\": rpc error: code = NotFound desc = could not find container \"333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4\": container with ID starting with 333e47dc9137e3dd89dbb6096487599d56811228520ff478962cd7e694d595d4 not found: ID does not exist"
Oct 31 20:07:51.982678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.982647    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"]
Oct 31 20:07:51.983043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.983025    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae4ce2-767b777bf4-dj85z"]
Oct 31 20:07:51.999167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:51.999131    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"]
Oct 31 20:07:52.001043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.001014    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c55cf1-676466dd6f-s8jbg"]
Oct 31 20:07:52.015818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.015784    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"]
Oct 31 20:07:52.025346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.025317    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c55cf1-6ff96ddc6c-np8m2"]
Oct 31 20:07:52.664631 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.664600    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420f78c4-3d9b-43b3-8115-f00fa70f956e" path="/var/lib/kubelet/pods/420f78c4-3d9b-43b3-8115-f00fa70f956e/volumes"
Oct 31 20:07:52.665036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.665015    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" path="/var/lib/kubelet/pods/619b2c9c-f018-491e-b7e8-dae9abd7ccdc/volumes"
Oct 31 20:07:52.665353 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.665340    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" path="/var/lib/kubelet/pods/b37734fa-f33f-4d92-b6ce-0a81b16291d0/volumes"
Oct 31 20:07:52.904949 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:52.904913    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:07:54.564872 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.564820    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs" podUID="b69cfbd1-5c8b-473e-8244-48b487c848a2" containerName="payload-processing" containerID="cri-o://a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2" gracePeriod=25
Oct 31 20:07:54.636015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.635971    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerName="payload-pre-processing" containerID="cri-o://ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682" gracePeriod=25
Oct 31 20:07:54.822519 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.822453    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 20:07:54.922067 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.922028    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr\" (UniqueName: \"kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr\") pod \"b69cfbd1-5c8b-473e-8244-48b487c848a2\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") "
Oct 31 20:07:54.922278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.922141    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume\") pod \"b69cfbd1-5c8b-473e-8244-48b487c848a2\" (UID: \"b69cfbd1-5c8b-473e-8244-48b487c848a2\") "
Oct 31 20:07:54.922555 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.922515    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume" pod "b69cfbd1-5c8b-473e-8244-48b487c848a2" (UID: "b69cfbd1-5c8b-473e-8244-48b487c848a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:54.922710 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.922694    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b69cfbd1-5c8b-473e-8244-48b487c848a2-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:54.924355 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.924331    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr" pod "b69cfbd1-5c8b-473e-8244-48b487c848a2" (UID: "b69cfbd1-5c8b-473e-8244-48b487c848a2"). InnerVolumeSpecName "kube-api-access-b9qbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:54.967712 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.967674    5958 generic.go:356] "Generic (PLEG): container finished" podID="b69cfbd1-5c8b-473e-8244-48b487c848a2" containerID="a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2" exitCode=0
Oct 31 20:07:54.967950 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.967724    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"
Oct 31 20:07:54.967950 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.967753    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs" event={"ID":"b69cfbd1-5c8b-473e-8244-48b487c848a2","Type":"ContainerDied","Data":"a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2"}
Oct 31 20:07:54.967950 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.967805    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs" event={"ID":"b69cfbd1-5c8b-473e-8244-48b487c848a2","Type":"ContainerDied","Data":"73376dcc9f9f3ddee6bee9240b9fc2b3e8e5f8517db08b5381e53a1e49c62adb"}
Oct 31 20:07:54.967950 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.967831    5958 scope.go:122] "RemoveContainer" containerID="a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2"
Oct 31 20:07:54.978152 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.978134    5958 scope.go:122] "RemoveContainer" containerID="a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2"
Oct 31 20:07:54.978422 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:07:54.978402    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2\": container with ID starting with a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2 not found: ID does not exist" containerID="a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2"
Oct 31 20:07:54.978490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.978430    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2"} err="failed to get container status \"a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2\": rpc error: code = NotFound desc = could not find container \"a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2\": container with ID starting with a91f1c64959758d570d1201dcc65a58e3157b2a1316a199423c8593fb0352db2 not found: ID does not exist"
Oct 31 20:07:54.998107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:54.998070    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"]
Oct 31 20:07:55.008064 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:55.008033    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae4ce2-75f79f6df7-h7nqs"]
Oct 31 20:07:55.023898 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:55.023867    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qbr\" (UniqueName: \"kubernetes.io/projected/b69cfbd1-5c8b-473e-8244-48b487c848a2-kube-api-access-b9qbr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:55.504288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:55.504251    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:56.612288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:56.612256    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"]
Oct 31 20:07:56.612792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:56.612481    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy" containerID="cri-o://6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b" gracePeriod=30
Oct 31 20:07:56.662867 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:56.662838    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69cfbd1-5c8b-473e-8244-48b487c848a2" path="/var/lib/kubelet/pods/b69cfbd1-5c8b-473e-8244-48b487c848a2/volumes"
Oct 31 20:07:57.280505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.280459    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"]
Oct 31 20:07:57.281162 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281139    5958 cpu_manager.go:414] "containerMap: removing container" podUID="128d8a3d-d166-482f-9e12-00ebf955e68b" containerName="maas-api"
Oct 31 20:07:57.281162 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281162    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128d8a3d-d166-482f-9e12-00ebf955e68b" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281183    5958 cpu_manager.go:414] "containerMap: removing container" podUID="799e4486-7297-46e5-8b27-30218f000e80" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281189    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799e4486-7297-46e5-8b27-30218f000e80" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281201    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5730e849-2fe7-41a5-894f-3def0ee02cb8" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281208    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5730e849-2fe7-41a5-894f-3def0ee02cb8" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281226    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5783b36a-c79f-47b7-a407-cbfaf48567c1" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281230    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5783b36a-c79f-47b7-a407-cbfaf48567c1" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281242    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281250    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281262    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281270    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281284    5958 cpu_manager.go:414] "containerMap: removing container" podUID="420f78c4-3d9b-43b3-8115-f00fa70f956e" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281289    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420f78c4-3d9b-43b3-8115-f00fa70f956e" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281297    5958 cpu_manager.go:414] "containerMap: removing container" podUID="79d8b407-edfa-4b0b-925d-ecaf2d014038" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281302    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d8b407-edfa-4b0b-925d-ecaf2d014038" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281307    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5a2738d1-4587-4dfd-8ea4-2542d0d9b842" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281311    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2738d1-4587-4dfd-8ea4-2542d0d9b842" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281324    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b69cfbd1-5c8b-473e-8244-48b487c848a2" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281329    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69cfbd1-5c8b-473e-8244-48b487c848a2" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281344    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3d88c070-0722-481d-9056-1f2aa89a02a3" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281351    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d88c070-0722-481d-9056-1f2aa89a02a3" containerName="maas-api"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281363    5958 cpu_manager.go:414] "containerMap: removing container" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281368    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerName="payload-pre-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281374    5958 cpu_manager.go:414] "containerMap: removing container" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281379    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281387    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerName="payload-processing"
Oct 31 20:07:57.281375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281392    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerName="payload-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281398    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281403    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281415    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dc60ec47-7d8b-4643-8b18-5288ac867803" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281422    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc60ec47-7d8b-4643-8b18-5288ac867803" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281507    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="420f78c4-3d9b-43b3-8115-f00fa70f956e" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281516    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d8b407-edfa-4b0b-925d-ecaf2d014038" containerName="payload-pre-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281523    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2738d1-4587-4dfd-8ea4-2542d0d9b842" containerName="payload-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281533    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69cfbd1-5c8b-473e-8244-48b487c848a2" containerName="payload-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281556    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d88c070-0722-481d-9056-1f2aa89a02a3" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281567    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="619b2c9c-f018-491e-b7e8-dae9abd7ccdc" containerName="payload-pre-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281574    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="072664a6-4f1d-4318-b9e4-b31190d80f64" containerName="payload-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281586    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37734fa-f33f-4d92-b6ce-0a81b16291d0" containerName="payload-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281591    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2140f87-1dde-4114-924f-7eb27f801109" containerName="istio-proxy"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281596    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc60ec47-7d8b-4643-8b18-5288ac867803" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281602    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="128d8a3d-d166-482f-9e12-00ebf955e68b" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281610    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="799e4486-7297-46e5-8b27-30218f000e80" containerName="payload-pre-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281620    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5730e849-2fe7-41a5-894f-3def0ee02cb8" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281631    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5783b36a-c79f-47b7-a407-cbfaf48567c1" containerName="maas-api"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281641    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae17531-6e12-4803-9977-cb2e9a349df5" containerName="payload-pre-processing"
Oct 31 20:07:57.283007 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.281650    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3a29c0-4c9f-4740-b379-d0d54dbb818e" containerName="payload-pre-processing"
Oct 31 20:07:57.286416 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.286395    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.303579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.303514    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"]
Oct 31 20:07:57.305068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.305022    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:57.305260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.305075    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:07:57.351764 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.351728    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.351948 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.351906    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng56r\" (UniqueName: \"kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.351998 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.351949    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.436728 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.436684    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" podUID="efbe721a-665b-4c89-b288-0d295946bba3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:57.436943 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.436752    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:57.453261 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.453223    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng56r\" (UniqueName: \"kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.453495 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.453292    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.453495 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.453358    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.453985 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.453959    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.455875 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.455843    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.467136 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.467113    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng56r\" (UniqueName: \"kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r\") pod \"maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.609571 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.609478    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:07:57.894361 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.894322    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"]
Oct 31 20:07:57.895361 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:07:57.895333    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3013073_4f99_4db1_b70d_c8728673cfc4.slice/crio-45a251223a5b1c6de16a17bc06151098a1b31e5e6e0c107dba9cd4f063c0168c WatchSource:0}: Error finding container 45a251223a5b1c6de16a17bc06151098a1b31e5e6e0c107dba9cd4f063c0168c: Status 404 returned error can't find the container with id 45a251223a5b1c6de16a17bc06151098a1b31e5e6e0c107dba9cd4f063c0168c
Oct 31 20:07:57.989668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:57.989603    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s" event={"ID":"e3013073-4f99-4db1-b70d-c8728673cfc4","Type":"ContainerStarted","Data":"45a251223a5b1c6de16a17bc06151098a1b31e5e6e0c107dba9cd4f063c0168c"}
Oct 31 20:07:58.997050 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:58.997013    5958 generic.go:356] "Generic (PLEG): container finished" podID="e3013073-4f99-4db1-b70d-c8728673cfc4" containerID="c1b5c0eb1f2a49a593e57e6aa44f6566237014e66062b1ddcf1ba9a3b2bcbb44" exitCode=0
Oct 31 20:07:58.997459 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:07:58.997098    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s" event={"ID":"e3013073-4f99-4db1-b70d-c8728673cfc4","Type":"ContainerDied","Data":"c1b5c0eb1f2a49a593e57e6aa44f6566237014e66062b1ddcf1ba9a3b2bcbb44"}
Oct 31 20:08:00.165724 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.165697    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:08:00.286862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.286759    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca\") pod \"e3013073-4f99-4db1-b70d-c8728673cfc4\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") "
Oct 31 20:08:00.287056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.286878    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r\" (UniqueName: \"kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r\") pod \"e3013073-4f99-4db1-b70d-c8728673cfc4\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") "
Oct 31 20:08:00.287056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.286970    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca\") pod \"e3013073-4f99-4db1-b70d-c8728673cfc4\" (UID: \"e3013073-4f99-4db1-b70d-c8728673cfc4\") "
Oct 31 20:08:00.287245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.287219    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca" pod "e3013073-4f99-4db1-b70d-c8728673cfc4" (UID: "e3013073-4f99-4db1-b70d-c8728673cfc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:00.287762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.287469    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:00.289195 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.289168    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca" pod "e3013073-4f99-4db1-b70d-c8728673cfc4" (UID: "e3013073-4f99-4db1-b70d-c8728673cfc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:00.289259 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.289240    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r" pod "e3013073-4f99-4db1-b70d-c8728673cfc4" (UID: "e3013073-4f99-4db1-b70d-c8728673cfc4"). InnerVolumeSpecName "kube-api-access-ng56r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:00.388715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.388681    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng56r\" (UniqueName: \"kubernetes.io/projected/e3013073-4f99-4db1-b70d-c8728673cfc4-kube-api-access-ng56r\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:00.388715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:00.388714    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3013073-4f99-4db1-b70d-c8728673cfc4-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:01.011211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.011115    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s" event={"ID":"e3013073-4f99-4db1-b70d-c8728673cfc4","Type":"ContainerDied","Data":"45a251223a5b1c6de16a17bc06151098a1b31e5e6e0c107dba9cd4f063c0168c"}
Oct 31 20:08:01.011211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.011177    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a251223a5b1c6de16a17bc06151098a1b31e5e6e0c107dba9cd4f063c0168c"
Oct 31 20:08:01.011211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.011129    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"
Oct 31 20:08:01.882962 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.882898    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.941344    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"]
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.942737    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e3013073-4f99-4db1-b70d-c8728673cfc4" containerName="revoke-keys"
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.942763    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3013073-4f99-4db1-b70d-c8728673cfc4" containerName="revoke-keys"
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.942817    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy"
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.942826    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy"
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.943181    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3013073-4f99-4db1-b70d-c8728673cfc4" containerName="revoke-keys"
Oct 31 20:08:01.945075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.943212    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" containerName="istio-proxy"
Oct 31 20:08:01.954912 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.954875    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:01.958445 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.958416    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"]
Oct 31 20:08:01.961696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.961661    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:08:01.970190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.970160    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:01.970414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:01.970358    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:08:02.008479 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008438    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.008676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008515    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.008676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008558    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.008818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008686    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.008818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008717    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.008818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008786    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.008818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008815    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.009037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008834    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:02.009037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008850    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.009037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008984    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy\") pod \"b1b11505-5149-4b74-b311-e6399e2925ec\" (UID: \"b1b11505-5149-4b74-b311-e6399e2925ec\") "
Oct 31 20:08:02.009037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.008938    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:02.009246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009022    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:02.009354 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009330    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:02.009433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009406    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:02.009584 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009529    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1b11505-5149-4b74-b311-e6399e2925ec-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.009584 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009576    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.009584 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009585    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.009815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009598    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.009815 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.009606    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.011258 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.011214    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:02.011258 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.011242    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:02.011446 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.011326    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:02.011632 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.011613    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9" pod "b1b11505-5149-4b74-b311-e6399e2925ec" (UID: "b1b11505-5149-4b74-b311-e6399e2925ec"). InnerVolumeSpecName "kube-api-access-6qwg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:02.019101 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.019074    5958 generic.go:356] "Generic (PLEG): container finished" podID="b1b11505-5149-4b74-b311-e6399e2925ec" containerID="6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b" exitCode=0
Oct 31 20:08:02.019246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.019137    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"
Oct 31 20:08:02.019246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.019149    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" event={"ID":"b1b11505-5149-4b74-b311-e6399e2925ec","Type":"ContainerDied","Data":"6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b"}
Oct 31 20:08:02.019246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.019201    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5" event={"ID":"b1b11505-5149-4b74-b311-e6399e2925ec","Type":"ContainerDied","Data":"4ea51d4e45e1ad3db780c6611ef265f87ca055481a5a3140b3a539dd12cf99da"}
Oct 31 20:08:02.019246 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.019222    5958 scope.go:122] "RemoveContainer" containerID="6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b"
Oct 31 20:08:02.037963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.037939    5958 scope.go:122] "RemoveContainer" containerID="6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b"
Oct 31 20:08:02.038318 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:02.038296    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b\": container with ID starting with 6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b not found: ID does not exist" containerID="6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b"
Oct 31 20:08:02.038427 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.038335    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b"} err="failed to get container status \"6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b\": rpc error: code = NotFound desc = could not find container \"6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b\": container with ID starting with 6bf024a4591c87e430799820942d81bca0beadd8a0e482bb5d98c37b4046680b not found: ID does not exist"
Oct 31 20:08:02.052961 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.052933    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"]
Oct 31 20:08:02.064047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.064017    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-gz4q5"]
Oct 31 20:08:02.071720 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.071686    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"]
Oct 31 20:08:02.110580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.110522    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.110816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.110753    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwjs\" (UniqueName: \"kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.110816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.110795    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.110990 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.110966    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwg9\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-kube-api-access-6qwg9\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.111056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.110997    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1b11505-5149-4b74-b311-e6399e2925ec-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.111120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.111083    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1b11505-5149-4b74-b311-e6399e2925ec-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.111120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.111100    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1b11505-5149-4b74-b311-e6399e2925ec-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.212248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.212156    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.212407 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.212260    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwjs\" (UniqueName: \"kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.212407 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.212280    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.212947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.212924    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.214589 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.214569    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.232048 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.232016    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwjs\" (UniqueName: \"kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs\") pod \"maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.272043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.272006    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:02.566210 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.566181    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"]
Oct 31 20:08:02.568132 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:08:02.568105    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d8d2b7_83c8_499b_8261_148adbc297a1.slice/crio-01f0116311b98f6c856ad52f3e7efa750bae05636ae82b6b6e6ffc95a3f3425f WatchSource:0}: Error finding container 01f0116311b98f6c856ad52f3e7efa750bae05636ae82b6b6e6ffc95a3f3425f: Status 404 returned error can't find the container with id 01f0116311b98f6c856ad52f3e7efa750bae05636ae82b6b6e6ffc95a3f3425f
Oct 31 20:08:02.664101 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:02.664070    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b11505-5149-4b74-b311-e6399e2925ec" path="/var/lib/kubelet/pods/b1b11505-5149-4b74-b311-e6399e2925ec/volumes"
Oct 31 20:08:03.026813 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:03.026770    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n" event={"ID":"f3d8d2b7-83c8-499b-8261-148adbc297a1","Type":"ContainerStarted","Data":"6b8528d46783794666425413d49bdef32d452a7783d8d9abca6d6dab1e3fd0a8"}
Oct 31 20:08:03.027314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:03.026835    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n" event={"ID":"f3d8d2b7-83c8-499b-8261-148adbc297a1","Type":"ContainerStarted","Data":"01f0116311b98f6c856ad52f3e7efa750bae05636ae82b6b6e6ffc95a3f3425f"}
Oct 31 20:08:03.048693 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:03.048637    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n" podStartSLOduration=2.048622192 podStartE2EDuration="2.048622192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:03.046892929 +0000 UTC m=+2876.508267545" watchObservedRunningTime="2026-10-31 20:08:03.048622192 +0000 UTC m=+2876.509996842"
Oct 31 20:08:04.036656 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:04.036615    5958 generic.go:356] "Generic (PLEG): container finished" podID="f3d8d2b7-83c8-499b-8261-148adbc297a1" containerID="6b8528d46783794666425413d49bdef32d452a7783d8d9abca6d6dab1e3fd0a8" exitCode=0
Oct 31 20:08:04.037078 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:04.036696    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n" event={"ID":"f3d8d2b7-83c8-499b-8261-148adbc297a1","Type":"ContainerDied","Data":"6b8528d46783794666425413d49bdef32d452a7783d8d9abca6d6dab1e3fd0a8"}
Oct 31 20:08:05.196323 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.196287    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:05.345569 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.345455    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca\") pod \"f3d8d2b7-83c8-499b-8261-148adbc297a1\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") "
Oct 31 20:08:05.345569 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.345497    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca\") pod \"f3d8d2b7-83c8-499b-8261-148adbc297a1\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") "
Oct 31 20:08:05.345800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.345648    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs\" (UniqueName: \"kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs\") pod \"f3d8d2b7-83c8-499b-8261-148adbc297a1\" (UID: \"f3d8d2b7-83c8-499b-8261-148adbc297a1\") "
Oct 31 20:08:05.345882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.345827    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca" pod "f3d8d2b7-83c8-499b-8261-148adbc297a1" (UID: "f3d8d2b7-83c8-499b-8261-148adbc297a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:05.346087 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.346066    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:05.347767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.347732    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca" pod "f3d8d2b7-83c8-499b-8261-148adbc297a1" (UID: "f3d8d2b7-83c8-499b-8261-148adbc297a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:05.347767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.347758    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs" pod "f3d8d2b7-83c8-499b-8261-148adbc297a1" (UID: "f3d8d2b7-83c8-499b-8261-148adbc297a1"). InnerVolumeSpecName "kube-api-access-pnwjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:05.446899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.446865    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnwjs\" (UniqueName: \"kubernetes.io/projected/f3d8d2b7-83c8-499b-8261-148adbc297a1-kube-api-access-pnwjs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:05.446899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.446899    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3d8d2b7-83c8-499b-8261-148adbc297a1-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:05.504501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:05.504457    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:06.060049 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:06.060018    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"
Oct 31 20:08:06.060243 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:06.060006    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n" event={"ID":"f3d8d2b7-83c8-499b-8261-148adbc297a1","Type":"ContainerDied","Data":"01f0116311b98f6c856ad52f3e7efa750bae05636ae82b6b6e6ffc95a3f3425f"}
Oct 31 20:08:06.060243 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:06.060175    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f0116311b98f6c856ad52f3e7efa750bae05636ae82b6b6e6ffc95a3f3425f"
Oct 31 20:08:06.808514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:06.808481    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"]
Oct 31 20:08:07.436895 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.436851    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:07.437090 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.436918    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" podUID="efbe721a-665b-4c89-b288-0d295946bba3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:07.482692 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.482656    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"]
Oct 31 20:08:07.497556 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.497496    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"]
Oct 31 20:08:07.641981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.641946    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"]
Oct 31 20:08:07.642709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.642683    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f3d8d2b7-83c8-499b-8261-148adbc297a1" containerName="revoke-keys"
Oct 31 20:08:07.642709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.642708    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d8d2b7-83c8-499b-8261-148adbc297a1" containerName="revoke-keys"
Oct 31 20:08:07.642880 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.642866    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d8d2b7-83c8-499b-8261-148adbc297a1" containerName="revoke-keys"
Oct 31 20:08:07.646169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.646154    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.656728 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.656702    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:07.657164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.656826    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:08:07.659989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.659959    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"]
Oct 31 20:08:07.775492 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.775404    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqgp\" (UniqueName: \"kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.775706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.775517    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.775706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.775562    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.876816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.876762    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqgp\" (UniqueName: \"kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.877297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.876832    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.877297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.876860    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.877725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.877704    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.879765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.879740    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.896825 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.896793    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqgp\" (UniqueName: \"kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp\") pod \"maas-api-revoke-sub-1b1e607a-nqmfk\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:07.964120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:07.964085    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:08.279906 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.279871    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"]
Oct 31 20:08:08.308976 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:08:08.308946    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcef7e2b_796e_4dc4_83dd_a6c50a91d894.slice/crio-cd4424bc3cde0b3187e4bb2b392ac616e83fa13975598db6817b266f92c98c09 WatchSource:0}: Error finding container cd4424bc3cde0b3187e4bb2b392ac616e83fa13975598db6817b266f92c98c09: Status 404 returned error can't find the container with id cd4424bc3cde0b3187e4bb2b392ac616e83fa13975598db6817b266f92c98c09
Oct 31 20:08:08.515136 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.515114    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 20:08:08.517266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.517247    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 20:08:08.686338 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686303    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume\") pod \"efbe721a-665b-4c89-b288-0d295946bba3\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") "
Oct 31 20:08:08.686338 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686348    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89\" (UniqueName: \"kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89\") pod \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") "
Oct 31 20:08:08.686633 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686385    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume\") pod \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\" (UID: \"f4b0ef0f-bc50-4197-a382-0c4df79be7ee\") "
Oct 31 20:08:08.686633 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686488    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq\" (UniqueName: \"kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq\") pod \"efbe721a-665b-4c89-b288-0d295946bba3\" (UID: \"efbe721a-665b-4c89-b288-0d295946bba3\") "
Oct 31 20:08:08.686816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686791    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume" pod "efbe721a-665b-4c89-b288-0d295946bba3" (UID: "efbe721a-665b-4c89-b288-0d295946bba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:08.686881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686829    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume" pod "f4b0ef0f-bc50-4197-a382-0c4df79be7ee" (UID: "f4b0ef0f-bc50-4197-a382-0c4df79be7ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:08.686984 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686970    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbe721a-665b-4c89-b288-0d295946bba3-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:08.687022 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.686990    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:08.688763 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.688732    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89" pod "f4b0ef0f-bc50-4197-a382-0c4df79be7ee" (UID: "f4b0ef0f-bc50-4197-a382-0c4df79be7ee"). InnerVolumeSpecName "kube-api-access-rzb89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:08.688883 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.688773    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq" pod "efbe721a-665b-4c89-b288-0d295946bba3" (UID: "efbe721a-665b-4c89-b288-0d295946bba3"). InnerVolumeSpecName "kube-api-access-dgmcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:08.788448 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.788400    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzb89\" (UniqueName: \"kubernetes.io/projected/f4b0ef0f-bc50-4197-a382-0c4df79be7ee-kube-api-access-rzb89\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:08.788448 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:08.788439    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmcq\" (UniqueName: \"kubernetes.io/projected/efbe721a-665b-4c89-b288-0d295946bba3-kube-api-access-dgmcq\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:09.079401 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.079361    5958 generic.go:356] "Generic (PLEG): container finished" podID="efbe721a-665b-4c89-b288-0d295946bba3" containerID="d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f" exitCode=0
Oct 31 20:08:09.079919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.079408    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"
Oct 31 20:08:09.079919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.079438    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" event={"ID":"efbe721a-665b-4c89-b288-0d295946bba3","Type":"ContainerDied","Data":"d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f"}
Oct 31 20:08:09.079919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.079489    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7" event={"ID":"efbe721a-665b-4c89-b288-0d295946bba3","Type":"ContainerDied","Data":"dc7aefabd163db05090cf5bf36905b13c0f0bcb6b0eafeb286265fee94fdf5cd"}
Oct 31 20:08:09.079919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.079514    5958 scope.go:122] "RemoveContainer" containerID="d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f"
Oct 31 20:08:09.081263 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.081236    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk" event={"ID":"fcef7e2b-796e-4dc4-83dd-a6c50a91d894","Type":"ContainerStarted","Data":"93d8cea57b366119ea846982eb67db7422c1d43d224ae43b37d2d5854af265de"}
Oct 31 20:08:09.081413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.081282    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk" event={"ID":"fcef7e2b-796e-4dc4-83dd-a6c50a91d894","Type":"ContainerStarted","Data":"cd4424bc3cde0b3187e4bb2b392ac616e83fa13975598db6817b266f92c98c09"}
Oct 31 20:08:09.082632 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.082609    5958 generic.go:356] "Generic (PLEG): container finished" podID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerID="af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c" exitCode=0
Oct 31 20:08:09.082762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.082653    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"
Oct 31 20:08:09.082762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.082689    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" event={"ID":"f4b0ef0f-bc50-4197-a382-0c4df79be7ee","Type":"ContainerDied","Data":"af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c"}
Oct 31 20:08:09.082762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.082725    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48" event={"ID":"f4b0ef0f-bc50-4197-a382-0c4df79be7ee","Type":"ContainerDied","Data":"4e9725542a167934f472f9c32fb6d99469a720a71411c4f13a5d3bf02548aaa3"}
Oct 31 20:08:09.091319 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.091303    5958 scope.go:122] "RemoveContainer" containerID="d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f"
Oct 31 20:08:09.091654 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:09.091632    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f\": container with ID starting with d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f not found: ID does not exist" containerID="d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f"
Oct 31 20:08:09.091758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.091660    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f"} err="failed to get container status \"d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f\": rpc error: code = NotFound desc = could not find container \"d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f\": container with ID starting with d69de6dfd61575b84aa4a8ad4283d82d5b65a9a4b91a7bca1b634e0286f05d1f not found: ID does not exist"
Oct 31 20:08:09.091758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.091690    5958 scope.go:122] "RemoveContainer" containerID="af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c"
Oct 31 20:08:09.100990 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.100970    5958 scope.go:122] "RemoveContainer" containerID="af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c"
Oct 31 20:08:09.101309 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:09.101288    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c\": container with ID starting with af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c not found: ID does not exist" containerID="af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c"
Oct 31 20:08:09.101359 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.101318    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c"} err="failed to get container status \"af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c\": rpc error: code = NotFound desc = could not find container \"af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c\": container with ID starting with af61d743b9a6458c0e7055b7ca0554c434a0b524a923f67ad411597a8516ae6c not found: ID does not exist"
Oct 31 20:08:09.120288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.120230    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk" podStartSLOduration=2.120216754 podStartE2EDuration="2.120216754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:09.118227259 +0000 UTC m=+2882.579601867" watchObservedRunningTime="2026-10-31 20:08:09.120216754 +0000 UTC m=+2882.581591403"
Oct 31 20:08:09.172938 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.172853    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"]
Oct 31 20:08:09.183135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.183090    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-435728-5fd9766559-wqr48"]
Oct 31 20:08:09.261649 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.261615    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"]
Oct 31 20:08:09.280608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:09.280577    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-435728-7b9576b5ff-xrkt7"]
Oct 31 20:08:10.090772 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:10.090738    5958 generic.go:356] "Generic (PLEG): container finished" podID="fcef7e2b-796e-4dc4-83dd-a6c50a91d894" containerID="93d8cea57b366119ea846982eb67db7422c1d43d224ae43b37d2d5854af265de" exitCode=0
Oct 31 20:08:10.091134 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:10.090817    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk" event={"ID":"fcef7e2b-796e-4dc4-83dd-a6c50a91d894","Type":"ContainerDied","Data":"93d8cea57b366119ea846982eb67db7422c1d43d224ae43b37d2d5854af265de"}
Oct 31 20:08:10.662736 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:10.662702    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbe721a-665b-4c89-b288-0d295946bba3" path="/var/lib/kubelet/pods/efbe721a-665b-4c89-b288-0d295946bba3/volumes"
Oct 31 20:08:10.663180 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:10.663165    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" path="/var/lib/kubelet/pods/f4b0ef0f-bc50-4197-a382-0c4df79be7ee/volumes"
Oct 31 20:08:11.251582 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.251535    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:11.418862 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.418822    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp\" (UniqueName: \"kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp\") pod \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") "
Oct 31 20:08:11.419037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.418952    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca\") pod \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") "
Oct 31 20:08:11.419037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.419016    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca\") pod \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\" (UID: \"fcef7e2b-796e-4dc4-83dd-a6c50a91d894\") "
Oct 31 20:08:11.419459 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.419429    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca" pod "fcef7e2b-796e-4dc4-83dd-a6c50a91d894" (UID: "fcef7e2b-796e-4dc4-83dd-a6c50a91d894"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:11.421272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.421238    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca" pod "fcef7e2b-796e-4dc4-83dd-a6c50a91d894" (UID: "fcef7e2b-796e-4dc4-83dd-a6c50a91d894"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:11.421366 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.421296    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp" pod "fcef7e2b-796e-4dc4-83dd-a6c50a91d894" (UID: "fcef7e2b-796e-4dc4-83dd-a6c50a91d894"). InnerVolumeSpecName "kube-api-access-4kqgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:11.520812 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.520771    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:11.520812 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.520804    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kqgp\" (UniqueName: \"kubernetes.io/projected/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-kube-api-access-4kqgp\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:11.520812 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:11.520814    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcef7e2b-796e-4dc4-83dd-a6c50a91d894-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:12.103807 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.103775    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"
Oct 31 20:08:12.104011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.103816    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk" event={"ID":"fcef7e2b-796e-4dc4-83dd-a6c50a91d894","Type":"ContainerDied","Data":"cd4424bc3cde0b3187e4bb2b392ac616e83fa13975598db6817b266f92c98c09"}
Oct 31 20:08:12.104011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.103874    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd4424bc3cde0b3187e4bb2b392ac616e83fa13975598db6817b266f92c98c09"
Oct 31 20:08:12.226052 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.226015    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"]
Oct 31 20:08:12.483839 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.483771    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerName="payload-processing" containerID="cri-o://42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628" gracePeriod=25
Oct 31 20:08:12.498353 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.498291    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerName="payload-pre-processing" containerID="cri-o://f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1" gracePeriod=25
Oct 31 20:08:12.583360 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583321    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"]
Oct 31 20:08:12.583928 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583894    5958 cpu_manager.go:414] "containerMap: removing container" podUID="efbe721a-665b-4c89-b288-0d295946bba3" containerName="payload-processing"
Oct 31 20:08:12.583928 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583919    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbe721a-665b-4c89-b288-0d295946bba3" containerName="payload-processing"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583935    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerName="payload-pre-processing"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583943    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerName="payload-pre-processing"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583963    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fcef7e2b-796e-4dc4-83dd-a6c50a91d894" containerName="revoke-keys"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.583970    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcef7e2b-796e-4dc4-83dd-a6c50a91d894" containerName="revoke-keys"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.584106    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbe721a-665b-4c89-b288-0d295946bba3" containerName="payload-processing"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.584116    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b0ef0f-bc50-4197-a382-0c4df79be7ee" containerName="payload-pre-processing"
Oct 31 20:08:12.584135 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.584127    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcef7e2b-796e-4dc4-83dd-a6c50a91d894" containerName="revoke-keys"
Oct 31 20:08:12.588263 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.588236    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.605237 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.605205    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"]
Oct 31 20:08:12.649431 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.649387    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:12.649692 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.649652    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:08:12.734773 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.734688    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.734773 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.734750    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pqv\" (UniqueName: \"kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.734985 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.734934    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.835808 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.835775    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.836024 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.835877    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pqv\" (UniqueName: \"kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.836098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.836033    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.836608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.836588    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.838414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.838383    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.859867 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.859828    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pqv\" (UniqueName: \"kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv\") pod \"maas-api-revoke-sub-67a0f62c-7ksr4\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:12.904588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:12.904527    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:13.192149 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:13.192121    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"]
Oct 31 20:08:13.193443 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:08:13.193403    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15e2576_4a06_4ffa_9d2c_ddc657e60da6.slice/crio-488692157c8051e74491aa134db960d7dcebc98b3ef1f990aded905c02dd764a WatchSource:0}: Error finding container 488692157c8051e74491aa134db960d7dcebc98b3ef1f990aded905c02dd764a: Status 404 returned error can't find the container with id 488692157c8051e74491aa134db960d7dcebc98b3ef1f990aded905c02dd764a
Oct 31 20:08:14.115723 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.115686    5958 generic.go:356] "Generic (PLEG): container finished" podID="a15e2576-4a06-4ffa-9d2c-ddc657e60da6" containerID="55d3325f7cf3b5561158e67c50a9cde7204f1960980934e206760d751ae9ccaf" exitCode=0
Oct 31 20:08:14.116266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.115812    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4" event={"ID":"a15e2576-4a06-4ffa-9d2c-ddc657e60da6","Type":"ContainerDied","Data":"55d3325f7cf3b5561158e67c50a9cde7204f1960980934e206760d751ae9ccaf"}
Oct 31 20:08:14.116266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.115869    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4" event={"ID":"a15e2576-4a06-4ffa-9d2c-ddc657e60da6","Type":"ContainerStarted","Data":"488692157c8051e74491aa134db960d7dcebc98b3ef1f990aded905c02dd764a"}
Oct 31 20:08:14.908006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.907934    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 20:08:14.962477 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.962441    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97\" (UniqueName: \"kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97\") pod \"ba25a2f1-c066-4891-a93c-0e20f9e74418\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") "
Oct 31 20:08:14.962670 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.962566    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume\") pod \"ba25a2f1-c066-4891-a93c-0e20f9e74418\" (UID: \"ba25a2f1-c066-4891-a93c-0e20f9e74418\") "
Oct 31 20:08:14.963067 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.963035    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume" pod "ba25a2f1-c066-4891-a93c-0e20f9e74418" (UID: "ba25a2f1-c066-4891-a93c-0e20f9e74418"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:14.964762 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:14.964724    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97" pod "ba25a2f1-c066-4891-a93c-0e20f9e74418" (UID: "ba25a2f1-c066-4891-a93c-0e20f9e74418"). InnerVolumeSpecName "kube-api-access-sms97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:15.063509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.063468    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sms97\" (UniqueName: \"kubernetes.io/projected/ba25a2f1-c066-4891-a93c-0e20f9e74418-kube-api-access-sms97\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:15.063509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.063500    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba25a2f1-c066-4891-a93c-0e20f9e74418-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:15.122934 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.122901    5958 generic.go:356] "Generic (PLEG): container finished" podID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerID="ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682" exitCode=0
Oct 31 20:08:15.123376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.122952    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"
Oct 31 20:08:15.123376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.122991    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" event={"ID":"ba25a2f1-c066-4891-a93c-0e20f9e74418","Type":"ContainerDied","Data":"ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682"}
Oct 31 20:08:15.123376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.123049    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd" event={"ID":"ba25a2f1-c066-4891-a93c-0e20f9e74418","Type":"ContainerDied","Data":"270ac188d60d314bdd791523aa73feed1acde369edf24d7d860b34e1713b64bf"}
Oct 31 20:08:15.123376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.123077    5958 scope.go:122] "RemoveContainer" containerID="ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682"
Oct 31 20:08:15.134621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.134518    5958 scope.go:122] "RemoveContainer" containerID="ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682"
Oct 31 20:08:15.134856 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:15.134836    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682\": container with ID starting with ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682 not found: ID does not exist" containerID="ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682"
Oct 31 20:08:15.134927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.134869    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682"} err="failed to get container status \"ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682\": rpc error: code = NotFound desc = could not find container \"ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682\": container with ID starting with ae500268e8cef0f1afed82e8631c11931d711c46fd8fd2a2b8ce0733fc815682 not found: ID does not exist"
Oct 31 20:08:15.148892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.148860    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"]
Oct 31 20:08:15.152616 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.152585    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae4ce2-5f87bc98dc-dzhpd"]
Oct 31 20:08:15.277278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.277251    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:15.366561 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.366522    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv\" (UniqueName: \"kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv\") pod \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") "
Oct 31 20:08:15.366768 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.366687    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca\") pod \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") "
Oct 31 20:08:15.366768 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.366752    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca\") pod \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\" (UID: \"a15e2576-4a06-4ffa-9d2c-ddc657e60da6\") "
Oct 31 20:08:15.367122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.367085    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca" pod "a15e2576-4a06-4ffa-9d2c-ddc657e60da6" (UID: "a15e2576-4a06-4ffa-9d2c-ddc657e60da6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:15.368695 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.368675    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv" pod "a15e2576-4a06-4ffa-9d2c-ddc657e60da6" (UID: "a15e2576-4a06-4ffa-9d2c-ddc657e60da6"). InnerVolumeSpecName "kube-api-access-95pqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:15.368810 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.368794    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca" pod "a15e2576-4a06-4ffa-9d2c-ddc657e60da6" (UID: "a15e2576-4a06-4ffa-9d2c-ddc657e60da6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:15.467877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.467797    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95pqv\" (UniqueName: \"kubernetes.io/projected/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-kube-api-access-95pqv\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:15.467877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.467825    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:15.467877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:15.467835    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15e2576-4a06-4ffa-9d2c-ddc657e60da6-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:16.130795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:16.130710    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4" event={"ID":"a15e2576-4a06-4ffa-9d2c-ddc657e60da6","Type":"ContainerDied","Data":"488692157c8051e74491aa134db960d7dcebc98b3ef1f990aded905c02dd764a"}
Oct 31 20:08:16.130795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:16.130769    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="488692157c8051e74491aa134db960d7dcebc98b3ef1f990aded905c02dd764a"
Oct 31 20:08:16.130795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:16.130728    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"
Oct 31 20:08:16.662733 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:16.662690    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" path="/var/lib/kubelet/pods/ba25a2f1-c066-4891-a93c-0e20f9e74418/volumes"
Oct 31 20:08:17.046081 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:17.045988    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:08:17.092924 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.092881    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" containerID="cri-o://bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea" gracePeriod=45
Oct 31 20:08:17.197790 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.197743    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": dial tcp 10.134.0.88:8000: connect: connection refused"
Oct 31 20:08:17.227297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.227240    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing" containerID="cri-o://222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4" gracePeriod=25
Oct 31 20:08:17.357120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.357097    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 20:08:17.381972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.381941    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs\") pod \"4a5c4855-d499-4812-99ec-da312af9e686\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") "
Oct 31 20:08:17.382155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.381991    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm\") pod \"4a5c4855-d499-4812-99ec-da312af9e686\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") "
Oct 31 20:08:17.382155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382053    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache\") pod \"4a5c4855-d499-4812-99ec-da312af9e686\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") "
Oct 31 20:08:17.382155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382076    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home\") pod \"4a5c4855-d499-4812-99ec-da312af9e686\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") "
Oct 31 20:08:17.382155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382098    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir\") pod \"4a5c4855-d499-4812-99ec-da312af9e686\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") "
Oct 31 20:08:17.382155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382137    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5\" (UniqueName: \"kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5\") pod \"4a5c4855-d499-4812-99ec-da312af9e686\" (UID: \"4a5c4855-d499-4812-99ec-da312af9e686\") "
Oct 31 20:08:17.382451 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382338    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home" pod "4a5c4855-d499-4812-99ec-da312af9e686" (UID: "4a5c4855-d499-4812-99ec-da312af9e686"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.382451 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382349    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache" pod "4a5c4855-d499-4812-99ec-da312af9e686" (UID: "4a5c4855-d499-4812-99ec-da312af9e686"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.382451 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382417    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir" pod "4a5c4855-d499-4812-99ec-da312af9e686" (UID: "4a5c4855-d499-4812-99ec-da312af9e686"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.382637 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382598    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.382637 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382617    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.382637 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.382632    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.384287 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.384266    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm" pod "4a5c4855-d499-4812-99ec-da312af9e686" (UID: "4a5c4855-d499-4812-99ec-da312af9e686"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.384375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.384351    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs" pod "4a5c4855-d499-4812-99ec-da312af9e686" (UID: "4a5c4855-d499-4812-99ec-da312af9e686"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:17.384442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.384425    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5" pod "4a5c4855-d499-4812-99ec-da312af9e686" (UID: "4a5c4855-d499-4812-99ec-da312af9e686"). InnerVolumeSpecName "kube-api-access-9sjp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:17.483313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.483272    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a5c4855-d499-4812-99ec-da312af9e686-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.483313 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.483315    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjp5\" (UniqueName: \"kubernetes.io/projected/4a5c4855-d499-4812-99ec-da312af9e686-kube-api-access-9sjp5\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.483571 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:17.483331    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a5c4855-d499-4812-99ec-da312af9e686-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:18.145654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.145614    5958 generic.go:356] "Generic (PLEG): container finished" podID="4a5c4855-d499-4812-99ec-da312af9e686" containerID="bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea" exitCode=0
Oct 31 20:08:18.145654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.145654    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"
Oct 31 20:08:18.145919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.145711    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" event={"ID":"4a5c4855-d499-4812-99ec-da312af9e686","Type":"ContainerDied","Data":"bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea"}
Oct 31 20:08:18.145919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.145751    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn" event={"ID":"4a5c4855-d499-4812-99ec-da312af9e686","Type":"ContainerDied","Data":"fed0e2976e15438544f6211231b65a6fa101cbb855160820e84f8c8d74d76a54"}
Oct 31 20:08:18.145919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.145772    5958 scope.go:122] "RemoveContainer" containerID="bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea"
Oct 31 20:08:18.156186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.156163    5958 scope.go:122] "RemoveContainer" containerID="bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea"
Oct 31 20:08:18.156506 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:18.156488    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea\": container with ID starting with bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea not found: ID does not exist" containerID="bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea"
Oct 31 20:08:18.156589 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.156519    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea"} err="failed to get container status \"bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea\": rpc error: code = NotFound desc = could not find container \"bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea\": container with ID starting with bb3cd16efcd2e544301df0b57a6c9805aba84f8887fa1e81a487ac2fc736ddea not found: ID does not exist"
Oct 31 20:08:18.188458 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.188427    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"]
Oct 31 20:08:18.193937 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.193910    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-2-w1-8b083a-kserve-67d646f857-d4wgn"]
Oct 31 20:08:18.378572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.378526    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"]
Oct 31 20:08:18.378973 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.378760    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd" podUID="aca2385e-77ad-4075-b0df-a86eeab06863" containerName="maas-api" containerID="cri-o://7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5" gracePeriod=30
Oct 31 20:08:18.656223 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.656162    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 20:08:18.662887 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.662858    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5c4855-d499-4812-99ec-da312af9e686" path="/var/lib/kubelet/pods/4a5c4855-d499-4812-99ec-da312af9e686/volumes"
Oct 31 20:08:18.688999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.688966    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"]
Oct 31 20:08:18.695279 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.695242    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8b083a-dqz8s"]
Oct 31 20:08:18.795009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.794970    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2\" (UniqueName: \"kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2\") pod \"aca2385e-77ad-4075-b0df-a86eeab06863\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") "
Oct 31 20:08:18.795009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.795016    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca\") pod \"aca2385e-77ad-4075-b0df-a86eeab06863\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") "
Oct 31 20:08:18.795309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.795121    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls\") pod \"aca2385e-77ad-4075-b0df-a86eeab06863\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") "
Oct 31 20:08:18.795309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.795190    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs\") pod \"aca2385e-77ad-4075-b0df-a86eeab06863\" (UID: \"aca2385e-77ad-4075-b0df-a86eeab06863\") "
Oct 31 20:08:18.795770 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.795525    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca" pod "aca2385e-77ad-4075-b0df-a86eeab06863" (UID: "aca2385e-77ad-4075-b0df-a86eeab06863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:18.797969 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.797925    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs" pod "aca2385e-77ad-4075-b0df-a86eeab06863" (UID: "aca2385e-77ad-4075-b0df-a86eeab06863"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:18.799231 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.799154    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2" pod "aca2385e-77ad-4075-b0df-a86eeab06863" (UID: "aca2385e-77ad-4075-b0df-a86eeab06863"). InnerVolumeSpecName "kube-api-access-f6ng2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:18.799426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.799334    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls" pod "aca2385e-77ad-4075-b0df-a86eeab06863" (UID: "aca2385e-77ad-4075-b0df-a86eeab06863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:18.896756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.896725    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:18.896756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.896754    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ng2\" (UniqueName: \"kubernetes.io/projected/aca2385e-77ad-4075-b0df-a86eeab06863-kube-api-access-f6ng2\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:18.896756 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.896765    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aca2385e-77ad-4075-b0df-a86eeab06863-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:18.897043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:18.896774    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aca2385e-77ad-4075-b0df-a86eeab06863-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:19.160586 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.160532    5958 generic.go:356] "Generic (PLEG): container finished" podID="aca2385e-77ad-4075-b0df-a86eeab06863" containerID="7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5" exitCode=0
Oct 31 20:08:19.160765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.160604    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"
Oct 31 20:08:19.160765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.160616    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd" event={"ID":"aca2385e-77ad-4075-b0df-a86eeab06863","Type":"ContainerDied","Data":"7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5"}
Oct 31 20:08:19.160765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.160692    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd" event={"ID":"aca2385e-77ad-4075-b0df-a86eeab06863","Type":"ContainerDied","Data":"9708cf3d087e0248208bf70c4fe2657361ce0f9490b426424cba0ec14fe99f22"}
Oct 31 20:08:19.160765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.160714    5958 scope.go:122] "RemoveContainer" containerID="7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5"
Oct 31 20:08:19.171120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.171098    5958 scope.go:122] "RemoveContainer" containerID="7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5"
Oct 31 20:08:19.171416 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:19.171396    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5\": container with ID starting with 7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5 not found: ID does not exist" containerID="7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5"
Oct 31 20:08:19.171477 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.171430    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5"} err="failed to get container status \"7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5\": rpc error: code = NotFound desc = could not find container \"7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5\": container with ID starting with 7795a4e36b3fcb2d137610bd97725578d1ca810288e1552490f55fee6f855ff5 not found: ID does not exist"
Oct 31 20:08:19.189965 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.189932    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"]
Oct 31 20:08:19.192901 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:19.192875    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-8b083a-6f97fb6898-cdtjd"]
Oct 31 20:08:20.664012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:20.663974    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca2385e-77ad-4075-b0df-a86eeab06863" path="/var/lib/kubelet/pods/aca2385e-77ad-4075-b0df-a86eeab06863/volumes"
Oct 31 20:08:20.664443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:20.664427    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3013073-4f99-4db1-b70d-c8728673cfc4" path="/var/lib/kubelet/pods/e3013073-4f99-4db1-b70d-c8728673cfc4/volumes"
Oct 31 20:08:21.420186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:21.420150    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:21.830006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:21.829901    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main" containerID="cri-o://24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272" gracePeriod=45
Oct 31 20:08:22.092750 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.092697    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 20:08:22.132827 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.132670    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir\") pod \"5b812a42-fb15-427c-a378-e852f22fc881\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") "
Oct 31 20:08:22.133199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.132741    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n\" (UniqueName: \"kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n\") pod \"5b812a42-fb15-427c-a378-e852f22fc881\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") "
Oct 31 20:08:22.133339 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.133266    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm\") pod \"5b812a42-fb15-427c-a378-e852f22fc881\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") "
Oct 31 20:08:22.133339 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.133312    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home\") pod \"5b812a42-fb15-427c-a378-e852f22fc881\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") "
Oct 31 20:08:22.133455 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.133370    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache\") pod \"5b812a42-fb15-427c-a378-e852f22fc881\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") "
Oct 31 20:08:22.133455 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.133413    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs\") pod \"5b812a42-fb15-427c-a378-e852f22fc881\" (UID: \"5b812a42-fb15-427c-a378-e852f22fc881\") "
Oct 31 20:08:22.134776 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.134309    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir" pod "5b812a42-fb15-427c-a378-e852f22fc881" (UID: "5b812a42-fb15-427c-a378-e852f22fc881"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:22.134776 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.134535    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home" pod "5b812a42-fb15-427c-a378-e852f22fc881" (UID: "5b812a42-fb15-427c-a378-e852f22fc881"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:22.134776 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.134746    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache" pod "5b812a42-fb15-427c-a378-e852f22fc881" (UID: "5b812a42-fb15-427c-a378-e852f22fc881"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:22.136943 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.136916    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs" pod "5b812a42-fb15-427c-a378-e852f22fc881" (UID: "5b812a42-fb15-427c-a378-e852f22fc881"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:22.137065 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.136994    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm" pod "5b812a42-fb15-427c-a378-e852f22fc881" (UID: "5b812a42-fb15-427c-a378-e852f22fc881"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:22.137899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.137872    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n" pod "5b812a42-fb15-427c-a378-e852f22fc881" (UID: "5b812a42-fb15-427c-a378-e852f22fc881"). InnerVolumeSpecName "kube-api-access-4q48n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:22.182014 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.181972    5958 generic.go:356] "Generic (PLEG): container finished" podID="5b812a42-fb15-427c-a378-e852f22fc881" containerID="24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272" exitCode=0
Oct 31 20:08:22.182190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.182015    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"
Oct 31 20:08:22.182190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.182025    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" event={"ID":"5b812a42-fb15-427c-a378-e852f22fc881","Type":"ContainerDied","Data":"24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272"}
Oct 31 20:08:22.182190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.182058    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g" event={"ID":"5b812a42-fb15-427c-a378-e852f22fc881","Type":"ContainerDied","Data":"336d95e71854cf866f0d1ec71d06b343c76c5235880797435d83a417fa75552a"}
Oct 31 20:08:22.182190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.182074    5958 scope.go:122] "RemoveContainer" containerID="24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272"
Oct 31 20:08:22.195191 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.195140    5958 scope.go:122] "RemoveContainer" containerID="24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272"
Oct 31 20:08:22.195733 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:22.195684    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272\": container with ID starting with 24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272 not found: ID does not exist" containerID="24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272"
Oct 31 20:08:22.195919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.195737    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272"} err="failed to get container status \"24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272\": rpc error: code = NotFound desc = could not find container \"24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272\": container with ID starting with 24a7b210f84349d0902f0edca6baf62b97af2e720bcdc622ea62362ce09bd272 not found: ID does not exist"
Oct 31 20:08:22.216737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.216699    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"]
Oct 31 20:08:22.219836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.219807    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/premium-w3-23814d-kserve-77cccbf599-nwt6g"]
Oct 31 20:08:22.235750 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.235710    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-tmp-dir\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:22.235750 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.235754    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q48n\" (UniqueName: \"kubernetes.io/projected/5b812a42-fb15-427c-a378-e852f22fc881-kube-api-access-4q48n\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:22.235970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.235770    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-dshm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:22.235970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.235786    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-home\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:22.235970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.235800    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b812a42-fb15-427c-a378-e852f22fc881-model-cache\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:22.235970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.235815    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b812a42-fb15-427c-a378-e852f22fc881-tls-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:22.426488 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.426447    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:22.663602 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:22.663566    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b812a42-fb15-427c-a378-e852f22fc881" path="/var/lib/kubelet/pods/5b812a42-fb15-427c-a378-e852f22fc881/volumes"
Oct 31 20:08:24.444110 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:24.444070    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:25.291619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:25.291573    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:08:25.836480 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:25.836431    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"]
Oct 31 20:08:25.846414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:25.846379    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-23814d-vx79n"]
Oct 31 20:08:26.663531 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:26.663495    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d8d2b7-83c8-499b-8261-148adbc297a1" path="/var/lib/kubelet/pods/f3d8d2b7-83c8-499b-8261-148adbc297a1/volumes"
Oct 31 20:08:31.419304 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:31.419261    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:32.426359 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.426311    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:32.756323 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.756295    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 20:08:32.796766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.796733    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 20:08:32.848422 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.848383    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm\" (UniqueName: \"kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm\") pod \"93581c47-2dc7-44c6-a6d0-78effaa42a43\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") "
Oct 31 20:08:32.848674 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.848432    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume\") pod \"93581c47-2dc7-44c6-a6d0-78effaa42a43\" (UID: \"93581c47-2dc7-44c6-a6d0-78effaa42a43\") "
Oct 31 20:08:32.848674 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.848470    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r\" (UniqueName: \"kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r\") pod \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") "
Oct 31 20:08:32.848674 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.848520    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume\") pod \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\" (UID: \"8642c922-5fb7-45c8-bdcf-99dea10faa6d\") "
Oct 31 20:08:32.848984 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.848959    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume" pod "93581c47-2dc7-44c6-a6d0-78effaa42a43" (UID: "93581c47-2dc7-44c6-a6d0-78effaa42a43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:32.849094 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.849015    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume" pod "8642c922-5fb7-45c8-bdcf-99dea10faa6d" (UID: "8642c922-5fb7-45c8-bdcf-99dea10faa6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:32.850666 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.850637    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r" pod "8642c922-5fb7-45c8-bdcf-99dea10faa6d" (UID: "8642c922-5fb7-45c8-bdcf-99dea10faa6d"). InnerVolumeSpecName "kube-api-access-9c25r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:32.850848 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.850684    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm" pod "93581c47-2dc7-44c6-a6d0-78effaa42a43" (UID: "93581c47-2dc7-44c6-a6d0-78effaa42a43"). InnerVolumeSpecName "kube-api-access-p6fwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:32.949891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.949851    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fwm\" (UniqueName: \"kubernetes.io/projected/93581c47-2dc7-44c6-a6d0-78effaa42a43-kube-api-access-p6fwm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:32.949891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.949882    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93581c47-2dc7-44c6-a6d0-78effaa42a43-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:32.949891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.949891    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c25r\" (UniqueName: \"kubernetes.io/projected/8642c922-5fb7-45c8-bdcf-99dea10faa6d-kube-api-access-9c25r\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:32.949891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:32.949903    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8642c922-5fb7-45c8-bdcf-99dea10faa6d-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.249580 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.249524    5958 generic.go:356] "Generic (PLEG): container finished" podID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerID="f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1" exitCode=0
Oct 31 20:08:33.249766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.249597    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"
Oct 31 20:08:33.249766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.249613    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" event={"ID":"93581c47-2dc7-44c6-a6d0-78effaa42a43","Type":"ContainerDied","Data":"f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1"}
Oct 31 20:08:33.249766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.249700    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz" event={"ID":"93581c47-2dc7-44c6-a6d0-78effaa42a43","Type":"ContainerDied","Data":"76598ef4af12325bcc055b96b8564ae4dfcf7c787ba0e59bbe13ec3ffa8c29bc"}
Oct 31 20:08:33.249766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.249730    5958 scope.go:122] "RemoveContainer" containerID="f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1"
Oct 31 20:08:33.251098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.251068    5958 generic.go:356] "Generic (PLEG): container finished" podID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerID="42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628" exitCode=0
Oct 31 20:08:33.251244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.251131    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"
Oct 31 20:08:33.251244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.251147    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" event={"ID":"8642c922-5fb7-45c8-bdcf-99dea10faa6d","Type":"ContainerDied","Data":"42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628"}
Oct 31 20:08:33.251244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.251194    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g" event={"ID":"8642c922-5fb7-45c8-bdcf-99dea10faa6d","Type":"ContainerDied","Data":"3b6756a9b29785619b65a3c2bb08069e84b40caebf88734fe937032f4dd341ec"}
Oct 31 20:08:33.261242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.261204    5958 scope.go:122] "RemoveContainer" containerID="f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1"
Oct 31 20:08:33.261629 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:33.261601    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1\": container with ID starting with f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1 not found: ID does not exist" containerID="f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1"
Oct 31 20:08:33.261714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.261640    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1"} err="failed to get container status \"f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1\": rpc error: code = NotFound desc = could not find container \"f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1\": container with ID starting with f3dc16c93ce5206a5450b348557cc5c9791141969cc961b82d7bd3e7eb10e5a1 not found: ID does not exist"
Oct 31 20:08:33.261714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.261663    5958 scope.go:122] "RemoveContainer" containerID="42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628"
Oct 31 20:08:33.270619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.270602    5958 scope.go:122] "RemoveContainer" containerID="42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628"
Oct 31 20:08:33.270925 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:33.270907    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628\": container with ID starting with 42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628 not found: ID does not exist" containerID="42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628"
Oct 31 20:08:33.270989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.270935    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628"} err="failed to get container status \"42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628\": rpc error: code = NotFound desc = could not find container \"42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628\": container with ID starting with 42f9cc609b3abe40d1ef83e6199eeeb59253f19e2c4cc5381bf7391cac1e8628 not found: ID does not exist"
Oct 31 20:08:33.277876 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.277849    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"]
Oct 31 20:08:33.283143 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.283114    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8b083a-76688799f5-29hdz"]
Oct 31 20:08:33.296339 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.296314    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"]
Oct 31 20:08:33.304503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.304481    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8b083a-67f5ddc8b5-pcl5g"]
Oct 31 20:08:33.913686 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:33.913649    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:08:34.030534 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:34.030501    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"]
Oct 31 20:08:34.030775 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:34.030742    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy" containerID="cri-o://3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14" gracePeriod=30
Oct 31 20:08:34.443896 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:34.443856    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:08:34.662197 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:34.662162    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" path="/var/lib/kubelet/pods/8642c922-5fb7-45c8-bdcf-99dea10faa6d/volumes"
Oct 31 20:08:34.662596 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:34.662578    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" path="/var/lib/kubelet/pods/93581c47-2dc7-44c6-a6d0-78effaa42a43/volumes"
Oct 31 20:08:36.713903 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:36.713872    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8689db56c-vdnn8"]
Oct 31 20:08:36.714574 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:36.714069    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8689db56c-vdnn8" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager" containerID="cri-o://281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d" gracePeriod=30
Oct 31 20:08:36.970891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:36.970828    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 20:08:37.095942 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.095911    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766\" (UniqueName: \"kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766\") pod \"4818c270-d27d-4689-9ea9-49d3b0e56579\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") "
Oct 31 20:08:37.096158 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.095986    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs\") pod \"4818c270-d27d-4689-9ea9-49d3b0e56579\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") "
Oct 31 20:08:37.096158 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.096067    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert\") pod \"4818c270-d27d-4689-9ea9-49d3b0e56579\" (UID: \"4818c270-d27d-4689-9ea9-49d3b0e56579\") "
Oct 31 20:08:37.097985 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.097952    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766" pod "4818c270-d27d-4689-9ea9-49d3b0e56579" (UID: "4818c270-d27d-4689-9ea9-49d3b0e56579"). InnerVolumeSpecName "kube-api-access-4j766". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:37.098117 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.098028    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert" pod "4818c270-d27d-4689-9ea9-49d3b0e56579" (UID: "4818c270-d27d-4689-9ea9-49d3b0e56579"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:37.098117 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.098079    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs" pod "4818c270-d27d-4689-9ea9-49d3b0e56579" (UID: "4818c270-d27d-4689-9ea9-49d3b0e56579"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:37.197757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.197707    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j766\" (UniqueName: \"kubernetes.io/projected/4818c270-d27d-4689-9ea9-49d3b0e56579-kube-api-access-4j766\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.197757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.197753    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.197757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.197763    5958 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4818c270-d27d-4689-9ea9-49d3b0e56579-webhook-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.279470 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.279422    5958 generic.go:356] "Generic (PLEG): container finished" podID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerID="281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d" exitCode=0
Oct 31 20:08:37.279470 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.279469    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-vdnn8" event={"ID":"4818c270-d27d-4689-9ea9-49d3b0e56579","Type":"ContainerDied","Data":"281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d"}
Oct 31 20:08:37.279760 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.279478    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-vdnn8"
Oct 31 20:08:37.279760 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.279526    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-vdnn8" event={"ID":"4818c270-d27d-4689-9ea9-49d3b0e56579","Type":"ContainerDied","Data":"a468e1667bedcf4047cdc35f51455286ed34fdbe392b1c30e937c18dc3d3b652"}
Oct 31 20:08:37.279760 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.279569    5958 scope.go:122] "RemoveContainer" containerID="281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d"
Oct 31 20:08:37.321450 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.321416    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8689db56c-vdnn8"]
Oct 31 20:08:37.326108 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.326082    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8689db56c-vdnn8"]
Oct 31 20:08:37.338354 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.338322    5958 scope.go:122] "RemoveContainer" containerID="24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"
Oct 31 20:08:37.346995 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.346973    5958 scope.go:122] "RemoveContainer" containerID="281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d"
Oct 31 20:08:37.347312 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:37.347289    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d\": container with ID starting with 281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d not found: ID does not exist" containerID="281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d"
Oct 31 20:08:37.347409 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.347321    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d"} err="failed to get container status \"281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d\": rpc error: code = NotFound desc = could not find container \"281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d\": container with ID starting with 281ad329358334d137783c653d36982fa6fcaf1b781634eccb0fa56e5ae4b54d not found: ID does not exist"
Oct 31 20:08:37.347409 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.347340    5958 scope.go:122] "RemoveContainer" containerID="24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"
Oct 31 20:08:37.347656 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:37.347630    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633\": container with ID starting with 24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633 not found: ID does not exist" containerID="24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"
Oct 31 20:08:37.347713 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.347664    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633"} err="failed to get container status \"24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633\": rpc error: code = NotFound desc = could not find container \"24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633\": container with ID starting with 24a9bcea32cea37e96e8dea16bca40d790ec7b21fa66d351d71897985fb7d633 not found: ID does not exist"
Oct 31 20:08:37.478616 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.478587    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 20:08:37.602489 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.602401    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76\") pod \"87ccdae5-03d1-421f-9c51-72765977005e\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") "
Oct 31 20:08:37.602489 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.602449    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume\") pod \"87ccdae5-03d1-421f-9c51-72765977005e\" (UID: \"87ccdae5-03d1-421f-9c51-72765977005e\") "
Oct 31 20:08:37.602904 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.602879    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume" pod "87ccdae5-03d1-421f-9c51-72765977005e" (UID: "87ccdae5-03d1-421f-9c51-72765977005e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:37.604605 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.604584    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76" pod "87ccdae5-03d1-421f-9c51-72765977005e" (UID: "87ccdae5-03d1-421f-9c51-72765977005e"). InnerVolumeSpecName "kube-api-access-dvc76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:37.682677 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.682641    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:08:37.703564 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.703521    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvc76\" (UniqueName: \"kubernetes.io/projected/87ccdae5-03d1-421f-9c51-72765977005e-kube-api-access-dvc76\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:37.703735 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:37.703586    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87ccdae5-03d1-421f-9c51-72765977005e-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:38.111671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.111636    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8689db56c-x4nnm"]
Oct 31 20:08:38.112088 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112075    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerName="payload-processing"
Oct 31 20:08:38.112133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112089    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerName="payload-processing"
Oct 31 20:08:38.112133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112105    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager"
Oct 31 20:08:38.112133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112111    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager"
Oct 31 20:08:38.112133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112121    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main"
Oct 31 20:08:38.112133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112127    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main"
Oct 31 20:08:38.112133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112134    5958 cpu_manager.go:414] "containerMap: removing container" podUID="aca2385e-77ad-4075-b0df-a86eeab06863" containerName="maas-api"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112139    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca2385e-77ad-4075-b0df-a86eeab06863" containerName="maas-api"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112146    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerName="payload-pre-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112152    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerName="payload-pre-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112169    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a15e2576-4a06-4ffa-9d2c-ddc657e60da6" containerName="revoke-keys"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112173    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15e2576-4a06-4ffa-9d2c-ddc657e60da6" containerName="revoke-keys"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112179    5958 cpu_manager.go:414] "containerMap: removing container" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerName="payload-pre-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112184    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerName="payload-pre-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112191    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112195    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112203    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112208    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112223    5958 cpu_manager.go:414] "containerMap: removing container" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112228    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112292    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112303    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ccdae5-03d1-421f-9c51-72765977005e" containerName="payload-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112311    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8642c922-5fb7-45c8-bdcf-99dea10faa6d" containerName="payload-processing"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112318    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" containerName="manager"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112323    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b812a42-fb15-427c-a378-e852f22fc881" containerName="main"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112330    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca2385e-77ad-4075-b0df-a86eeab06863" containerName="maas-api"
Oct 31 20:08:38.112330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112335    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba25a2f1-c066-4891-a93c-0e20f9e74418" containerName="payload-pre-processing"
Oct 31 20:08:38.113025 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112342    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15e2576-4a06-4ffa-9d2c-ddc657e60da6" containerName="revoke-keys"
Oct 31 20:08:38.113025 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112347    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="93581c47-2dc7-44c6-a6d0-78effaa42a43" containerName="payload-pre-processing"
Oct 31 20:08:38.113025 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.112352    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5c4855-d499-4812-99ec-da312af9e686" containerName="main"
Oct 31 20:08:38.115728 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.115709    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.124797 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.124682    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 20:08:38.127925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.127898    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689db56c-x4nnm"]
Oct 31 20:08:38.128501 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.128479    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 20:08:38.128640 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.128576    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 20:08:38.128866 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.128843    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v94n9\""
Oct 31 20:08:38.209284 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.209244    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.209514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.209364    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2568\" (UniqueName: \"kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.209514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.209432    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.288158 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.288122    5958 generic.go:356] "Generic (PLEG): container finished" podID="87ccdae5-03d1-421f-9c51-72765977005e" containerID="222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4" exitCode=0
Oct 31 20:08:38.288370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.288193    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" event={"ID":"87ccdae5-03d1-421f-9c51-72765977005e","Type":"ContainerDied","Data":"222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4"}
Oct 31 20:08:38.288370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.288204    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"
Oct 31 20:08:38.288370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.288251    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98" event={"ID":"87ccdae5-03d1-421f-9c51-72765977005e","Type":"ContainerDied","Data":"eaacb96052ad11324d2fb37bcc2a8c019d41b5893adb4271938df4bfab966f1c"}
Oct 31 20:08:38.288370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.288281    5958 scope.go:122] "RemoveContainer" containerID="222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4"
Oct 31 20:08:38.299685 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.299664    5958 scope.go:122] "RemoveContainer" containerID="222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4"
Oct 31 20:08:38.300040 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:38.300010    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4\": container with ID starting with 222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4 not found: ID does not exist" containerID="222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4"
Oct 31 20:08:38.300180 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.300051    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4"} err="failed to get container status \"222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4\": rpc error: code = NotFound desc = could not find container \"222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4\": container with ID starting with 222f523afd441f3298e5cf2f438980c404302fbe284dd8b6114b056ae6cb83a4 not found: ID does not exist"
Oct 31 20:08:38.310989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.310960    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2568\" (UniqueName: \"kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.311120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.311002    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.311120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.311058    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.313696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.313669    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.313696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.313685    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.318821 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.318796    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"]
Oct 31 20:08:38.324117 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.324089    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23814d-6cb8b9dc85-scg98"]
Oct 31 20:08:38.325556 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.325518    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2568\" (UniqueName: \"kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568\") pod \"maas-controller-8689db56c-x4nnm\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") " pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.435293 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.435253    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:38.663482 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.663437    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4818c270-d27d-4689-9ea9-49d3b0e56579" path="/var/lib/kubelet/pods/4818c270-d27d-4689-9ea9-49d3b0e56579/volumes"
Oct 31 20:08:38.663973 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.663952    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ccdae5-03d1-421f-9c51-72765977005e" path="/var/lib/kubelet/pods/87ccdae5-03d1-421f-9c51-72765977005e/volumes"
Oct 31 20:08:38.724963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:38.724937    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689db56c-x4nnm"]
Oct 31 20:08:38.727849 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:08:38.727820    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3118718a_9c69_4e7e_8105_60d34c7dfd0d.slice/crio-80e908717e8b662008b1f5b527b32dba199e0998141824aca2f7f919be355564 WatchSource:0}: Error finding container 80e908717e8b662008b1f5b527b32dba199e0998141824aca2f7f919be355564: Status 404 returned error can't find the container with id 80e908717e8b662008b1f5b527b32dba199e0998141824aca2f7f919be355564
Oct 31 20:08:39.126214 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:39.126166    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34092465_27ca_4013_a8e8_1134623d0f4e.slice/crio-conmon-3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34092465_27ca_4013_a8e8_1134623d0f4e.slice/crio-3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:08:39.303496 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.303053    5958 generic.go:356] "Generic (PLEG): container finished" podID="34092465-27ca-4013-a8e8-1134623d0f4e" containerID="3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14" exitCode=0
Oct 31 20:08:39.303496 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.303249    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" event={"ID":"34092465-27ca-4013-a8e8-1134623d0f4e","Type":"ContainerDied","Data":"3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14"}
Oct 31 20:08:39.305310 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.305265    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-x4nnm" event={"ID":"3118718a-9c69-4e7e-8105-60d34c7dfd0d","Type":"ContainerStarted","Data":"80e908717e8b662008b1f5b527b32dba199e0998141824aca2f7f919be355564"}
Oct 31 20:08:39.475237 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.475204    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.660788    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.660879    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.660943    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.660977    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.661016    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.661046    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.661101    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.661147    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.661925 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.661184    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr\") pod \"34092465-27ca-4013-a8e8-1134623d0f4e\" (UID: \"34092465-27ca-4013-a8e8-1134623d0f4e\") "
Oct 31 20:08:39.665046 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.664763    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.668419 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.668389    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:39.669296 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.669256    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.670766 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.670591    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.671182 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.671116    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "kube-api-access-r24qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:39.671450 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.671412    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.674144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.672162    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:39.674144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.673833    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:39.674144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.673868    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy" pod "34092465-27ca-4013-a8e8-1134623d0f4e" (UID: "34092465-27ca-4013-a8e8-1134623d0f4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.762624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762586    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34092465-27ca-4013-a8e8-1134623d0f4e-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762622    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762635    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34092465-27ca-4013-a8e8-1134623d0f4e-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762648    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24qr\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-kube-api-access-r24qr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762663    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762677    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34092465-27ca-4013-a8e8-1134623d0f4e-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762688    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762699    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.762908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:39.762712    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34092465-27ca-4013-a8e8-1134623d0f4e-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:40.329215 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.328170    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-x4nnm" event={"ID":"3118718a-9c69-4e7e-8105-60d34c7dfd0d","Type":"ContainerStarted","Data":"f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860"}
Oct 31 20:08:40.329215 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.329154    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:40.332049 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.332024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868" event={"ID":"34092465-27ca-4013-a8e8-1134623d0f4e","Type":"ContainerDied","Data":"11c455bf4c59e234c48d329afdcdab75cce6ddc2e1652d9f7e3aa0ede10e9c45"}
Oct 31 20:08:40.332293 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.332266    5958 scope.go:122] "RemoveContainer" containerID="3827de64b2f7fb4627d2e2034b19a1ecfe0f061443e875c7a5b69d7b6b40ab14"
Oct 31 20:08:40.332570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.332533    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"
Oct 31 20:08:40.486976 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.486917    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8689db56c-x4nnm" podStartSLOduration=1.850974425 podStartE2EDuration="2.486900949s" totalImagesPullingTime="635.926524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:40.483901109 +0000 UTC m=+2913.945275724" watchObservedRunningTime="2026-10-31 20:08:40.486900949 +0000 UTC m=+2913.948275563"
Oct 31 20:08:40.525913 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.525871    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"]
Oct 31 20:08:40.539167 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.539091    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bn868"]
Oct 31 20:08:40.673073 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:40.673034    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" path="/var/lib/kubelet/pods/34092465-27ca-4013-a8e8-1134623d0f4e/volumes"
Oct 31 20:08:47.045909 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:08:47.045858    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:08:52.347453 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:52.347423    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:08:57.075683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.075648    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"]
Oct 31 20:08:57.076191 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.076103    5958 cpu_manager.go:414] "containerMap: removing container" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy"
Oct 31 20:08:57.076191 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.076114    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy"
Oct 31 20:08:57.076279 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.076198    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="34092465-27ca-4013-a8e8-1134623d0f4e" containerName="istio-proxy"
Oct 31 20:08:57.079334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.079317    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.089317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.089261    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:57.089317 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.089269    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:08:57.091608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.091583    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"]
Oct 31 20:08:57.147870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.147834    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6hr\" (UniqueName: \"kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.148089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.147909    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.148089 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.148017    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.249613 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.249573    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.249805 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.249642    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.249805 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.249766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hr\" (UniqueName: \"kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.250265 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.250239    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.251939 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.251918    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.264052 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.264020    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6hr\" (UniqueName: \"kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr\") pod \"maas-api-revoke-sub-85237e45-7fbzl\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.393927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.393890    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:08:57.687757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:57.687729    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"]
Oct 31 20:08:57.690301 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:08:57.690273    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3ba027_cc50_4ccb_b674_72b52871b505.slice/crio-620e661a122917ed5352bef1d79036bb9a08083d94d13e5524e7967c0f2cba83 WatchSource:0}: Error finding container 620e661a122917ed5352bef1d79036bb9a08083d94d13e5524e7967c0f2cba83: Status 404 returned error can't find the container with id 620e661a122917ed5352bef1d79036bb9a08083d94d13e5524e7967c0f2cba83
Oct 31 20:08:58.438178 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:58.438140    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl" event={"ID":"3f3ba027-cc50-4ccb-b674-72b52871b505","Type":"ContainerStarted","Data":"329c1de7127c3a12413ab4e20d223c39c882f46ed0bcc5c463fb64f7acce331a"}
Oct 31 20:08:58.438631 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:58.438203    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl" event={"ID":"3f3ba027-cc50-4ccb-b674-72b52871b505","Type":"ContainerStarted","Data":"620e661a122917ed5352bef1d79036bb9a08083d94d13e5524e7967c0f2cba83"}
Oct 31 20:08:58.460475 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:58.460420    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl" podStartSLOduration=1.460404194 podStartE2EDuration="1.460404194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:58.459668653 +0000 UTC m=+2931.921043269" watchObservedRunningTime="2026-10-31 20:08:58.460404194 +0000 UTC m=+2931.921778809"
Oct 31 20:08:59.020142 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:59.020102    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:08:59.445354 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:59.445313    5958 generic.go:356] "Generic (PLEG): container finished" podID="3f3ba027-cc50-4ccb-b674-72b52871b505" containerID="329c1de7127c3a12413ab4e20d223c39c882f46ed0bcc5c463fb64f7acce331a" exitCode=0
Oct 31 20:08:59.445906 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:08:59.445396    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl" event={"ID":"3f3ba027-cc50-4ccb-b674-72b52871b505","Type":"ContainerDied","Data":"329c1de7127c3a12413ab4e20d223c39c882f46ed0bcc5c463fb64f7acce331a"}
Oct 31 20:09:00.614424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.614395    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:09:00.686310 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.686278    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca\") pod \"3f3ba027-cc50-4ccb-b674-72b52871b505\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") "
Oct 31 20:09:00.686530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.686401    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca\") pod \"3f3ba027-cc50-4ccb-b674-72b52871b505\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") "
Oct 31 20:09:00.686530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.686427    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr\" (UniqueName: \"kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr\") pod \"3f3ba027-cc50-4ccb-b674-72b52871b505\" (UID: \"3f3ba027-cc50-4ccb-b674-72b52871b505\") "
Oct 31 20:09:00.686863 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.686827    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca" pod "3f3ba027-cc50-4ccb-b674-72b52871b505" (UID: "3f3ba027-cc50-4ccb-b674-72b52871b505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:09:00.688505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.688481    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca" pod "3f3ba027-cc50-4ccb-b674-72b52871b505" (UID: "3f3ba027-cc50-4ccb-b674-72b52871b505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:09:00.688631 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.688481    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr" pod "3f3ba027-cc50-4ccb-b674-72b52871b505" (UID: "3f3ba027-cc50-4ccb-b674-72b52871b505"). InnerVolumeSpecName "kube-api-access-dt6hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:00.787709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.787612    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:00.787709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.787649    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3ba027-cc50-4ccb-b674-72b52871b505-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:00.787709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:00.787663    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6hr\" (UniqueName: \"kubernetes.io/projected/3f3ba027-cc50-4ccb-b674-72b52871b505-kube-api-access-dt6hr\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:01.458159 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:01.458131    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"
Oct 31 20:09:01.458159 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:01.458146    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl" event={"ID":"3f3ba027-cc50-4ccb-b674-72b52871b505","Type":"ContainerDied","Data":"620e661a122917ed5352bef1d79036bb9a08083d94d13e5524e7967c0f2cba83"}
Oct 31 20:09:01.458489 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:01.458192    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="620e661a122917ed5352bef1d79036bb9a08083d94d13e5524e7967c0f2cba83"
Oct 31 20:09:08.666774 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:08.666741    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8689db56c-x4nnm"]
Oct 31 20:09:08.667260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:08.666923    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8689db56c-x4nnm" podUID="3118718a-9c69-4e7e-8105-60d34c7dfd0d" containerName="manager" containerID="cri-o://f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860" gracePeriod=30
Oct 31 20:09:08.924154 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:08.924085    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:09:09.075847 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.075814    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568\" (UniqueName: \"kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568\") pod \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") "
Oct 31 20:09:09.076040 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.075874    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs\") pod \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") "
Oct 31 20:09:09.076040 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.075915    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert\") pod \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\" (UID: \"3118718a-9c69-4e7e-8105-60d34c7dfd0d\") "
Oct 31 20:09:09.078099 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.078061    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568" pod "3118718a-9c69-4e7e-8105-60d34c7dfd0d" (UID: "3118718a-9c69-4e7e-8105-60d34c7dfd0d"). InnerVolumeSpecName "kube-api-access-q2568". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:09.078237 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.078090    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs" pod "3118718a-9c69-4e7e-8105-60d34c7dfd0d" (UID: "3118718a-9c69-4e7e-8105-60d34c7dfd0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:09:09.078237 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.078144    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert" pod "3118718a-9c69-4e7e-8105-60d34c7dfd0d" (UID: "3118718a-9c69-4e7e-8105-60d34c7dfd0d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:09:09.177553 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.177461    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2568\" (UniqueName: \"kubernetes.io/projected/3118718a-9c69-4e7e-8105-60d34c7dfd0d-kube-api-access-q2568\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:09.177553 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.177493    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:09.177553 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.177502    5958 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3118718a-9c69-4e7e-8105-60d34c7dfd0d-webhook-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:09.503066 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.502970    5958 generic.go:356] "Generic (PLEG): container finished" podID="3118718a-9c69-4e7e-8105-60d34c7dfd0d" containerID="f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860" exitCode=0
Oct 31 20:09:09.503066 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.503023    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-x4nnm"
Oct 31 20:09:09.503312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.503058    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-x4nnm" event={"ID":"3118718a-9c69-4e7e-8105-60d34c7dfd0d","Type":"ContainerDied","Data":"f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860"}
Oct 31 20:09:09.503312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.503129    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-x4nnm" event={"ID":"3118718a-9c69-4e7e-8105-60d34c7dfd0d","Type":"ContainerDied","Data":"80e908717e8b662008b1f5b527b32dba199e0998141824aca2f7f919be355564"}
Oct 31 20:09:09.503312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.503161    5958 scope.go:122] "RemoveContainer" containerID="f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860"
Oct 31 20:09:09.513384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.513363    5958 scope.go:122] "RemoveContainer" containerID="f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860"
Oct 31 20:09:09.513707 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:09:09.513684    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860\": container with ID starting with f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860 not found: ID does not exist" containerID="f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860"
Oct 31 20:09:09.513819 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.513720    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860"} err="failed to get container status \"f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860\": rpc error: code = NotFound desc = could not find container \"f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860\": container with ID starting with f954f8ec1cf51aae13f177d3cab5f190f9f360fa4e729b09eddc0c55d4631860 not found: ID does not exist"
Oct 31 20:09:09.527316 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.527285    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8689db56c-x4nnm"]
Oct 31 20:09:09.532665 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:09.532637    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8689db56c-x4nnm"]
Oct 31 20:09:10.662041 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.662009    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3118718a-9c69-4e7e-8105-60d34c7dfd0d" path="/var/lib/kubelet/pods/3118718a-9c69-4e7e-8105-60d34c7dfd0d/volumes"
Oct 31 20:09:10.830303 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830267    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8689db56c-5m4t5"]
Oct 31 20:09:10.830743 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830730    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3118718a-9c69-4e7e-8105-60d34c7dfd0d" containerName="manager"
Oct 31 20:09:10.830849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830744    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3118718a-9c69-4e7e-8105-60d34c7dfd0d" containerName="manager"
Oct 31 20:09:10.830849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830763    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3f3ba027-cc50-4ccb-b674-72b52871b505" containerName="revoke-keys"
Oct 31 20:09:10.830849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830768    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3ba027-cc50-4ccb-b674-72b52871b505" containerName="revoke-keys"
Oct 31 20:09:10.830849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830839    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3118718a-9c69-4e7e-8105-60d34c7dfd0d" containerName="manager"
Oct 31 20:09:10.830849 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.830849    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3ba027-cc50-4ccb-b674-72b52871b505" containerName="revoke-keys"
Oct 31 20:09:10.836056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.836025    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:10.845122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.845098    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 20:09:10.851518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.851494    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689db56c-5m4t5"]
Oct 31 20:09:10.852368 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.852348    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 20:09:10.852474 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.852461    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 20:09:10.852648 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.852629    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v94n9\""
Oct 31 20:09:10.996619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.996505    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0bc7265-4705-42fc-9e36-6497af8933e3-webhook-cert\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:10.996795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.996687    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kggl\" (UniqueName: \"kubernetes.io/projected/d0bc7265-4705-42fc-9e36-6497af8933e3-kube-api-access-2kggl\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:10.996795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:10.996727    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bc7265-4705-42fc-9e36-6497af8933e3-metrics-certs\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.097721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.097663    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kggl\" (UniqueName: \"kubernetes.io/projected/d0bc7265-4705-42fc-9e36-6497af8933e3-kube-api-access-2kggl\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.097721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.097728    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bc7265-4705-42fc-9e36-6497af8933e3-metrics-certs\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.097998 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.097783    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0bc7265-4705-42fc-9e36-6497af8933e3-webhook-cert\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.100366 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.100327    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bc7265-4705-42fc-9e36-6497af8933e3-metrics-certs\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.100499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.100379    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0bc7265-4705-42fc-9e36-6497af8933e3-webhook-cert\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.111370 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.111343    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kggl\" (UniqueName: \"kubernetes.io/projected/d0bc7265-4705-42fc-9e36-6497af8933e3-kube-api-access-2kggl\") pod \"maas-controller-8689db56c-5m4t5\" (UID: \"d0bc7265-4705-42fc-9e36-6497af8933e3\") " pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.149981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.149947    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:11.439860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.439831    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8689db56c-5m4t5"]
Oct 31 20:09:11.441740 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:09:11.441719    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bc7265_4705_42fc_9e36_6497af8933e3.slice/crio-303e45d1bc04239334d71672bd1918362320d0753b784fae908964c8c2472f87 WatchSource:0}: Error finding container 303e45d1bc04239334d71672bd1918362320d0753b784fae908964c8c2472f87: Status 404 returned error can't find the container with id 303e45d1bc04239334d71672bd1918362320d0753b784fae908964c8c2472f87
Oct 31 20:09:11.514955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:11.514917    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-5m4t5" event={"ID":"d0bc7265-4705-42fc-9e36-6497af8933e3","Type":"ContainerStarted","Data":"303e45d1bc04239334d71672bd1918362320d0753b784fae908964c8c2472f87"}
Oct 31 20:09:12.521837 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:12.521787    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8689db56c-5m4t5" event={"ID":"d0bc7265-4705-42fc-9e36-6497af8933e3","Type":"ContainerStarted","Data":"c4c9a29e416c065c2ad0b76d2ea38ba5a1656bc8ecc7af63d5f6f73ab07bf0a4"}
Oct 31 20:09:12.522634 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:12.521889    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:12.548169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:12.548110    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8689db56c-5m4t5" podStartSLOduration=2.072173784 podStartE2EDuration="2.548096098s" totalImagesPullingTime="475.922314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:12.547981773 +0000 UTC m=+2946.009356389" watchObservedRunningTime="2026-10-31 20:09:12.548096098 +0000 UTC m=+2946.009470712"
Oct 31 20:09:17.050693 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:09:17.050621    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:09:23.530195 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:23.530164    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8689db56c-5m4t5"
Oct 31 20:09:24.323324 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.323288    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"]
Oct 31 20:09:24.327180 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.327163    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.345985 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.345959    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"]
Oct 31 20:09:24.428926 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.428883    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429096 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.428932    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429096 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429020    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429096 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429055    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429096 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429072    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429184    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429216    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8wn\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429346 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429322    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.429446 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.429429    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.530810 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530769    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.530810 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530817    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530853    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530881    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530940    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530963    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.530999    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531024    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8wn\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531224    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531284    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531293    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531971 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531389    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531971 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531462    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.531971 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.531902    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.533375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.533357    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.533643 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.533623    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.545047 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.545017    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.545710 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.545668    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8wn\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn\") pod \"maas-default-gateway-openshift-default-75d7c785f-6ncmf\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.641709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.641674    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:24.923439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.923414    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"]
Oct 31 20:09:24.925734 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:09:24.925701    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2070b9bd_6ab1_4d9f_be48_3200efc1b85e.slice/crio-a8ee374b6def543bf8d99c22e97662218ba552ffa3e7b11f806d2029d3cc4b8e WatchSource:0}: Error finding container a8ee374b6def543bf8d99c22e97662218ba552ffa3e7b11f806d2029d3cc4b8e: Status 404 returned error can't find the container with id a8ee374b6def543bf8d99c22e97662218ba552ffa3e7b11f806d2029d3cc4b8e
Oct 31 20:09:24.927869 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.927836    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:09:24.927962 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.927900    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:09:24.927962 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:24.927928    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:09:25.591617 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:25.591583    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" event={"ID":"2070b9bd-6ab1-4d9f-be48-3200efc1b85e","Type":"ContainerStarted","Data":"8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88"}
Oct 31 20:09:25.592132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:25.591648    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" event={"ID":"2070b9bd-6ab1-4d9f-be48-3200efc1b85e","Type":"ContainerStarted","Data":"a8ee374b6def543bf8d99c22e97662218ba552ffa3e7b11f806d2029d3cc4b8e"}
Oct 31 20:09:25.627478 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:25.627402    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podStartSLOduration=1.6273671589999998 podStartE2EDuration="1.627367159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:25.620259242 +0000 UTC m=+2959.081633862" watchObservedRunningTime="2026-10-31 20:09:25.627367159 +0000 UTC m=+2959.088741775"
Oct 31 20:09:25.642562 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:25.642506    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:25.644036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:25.644000    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:25.644178 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:25.644054    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:26.642211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:26.642173    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:26.642672 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:26.642226    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:27.643124 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:27.643089    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:27.643566 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:27.643157    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:28.642071 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:28.642031    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:28.642362 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:28.642102    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:29.642498 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:29.642457    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:29.642911 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:29.642537    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:30.642605 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:30.642567    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:30.643017 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:30.642622    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:31.642242 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:31.642206    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6ncmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Oct 31 20:09:31.642449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:31.642275    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Oct 31 20:09:33.183012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:33.182968    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:33.183556 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:33.183338    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:33.184077 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:33.184056    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:09:45.619615 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:45.619578    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:09:47.100901 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:09:47.100820    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:09:50.346561 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.346517    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"]
Oct 31 20:09:50.350585 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.350535    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.359228 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.358885    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"]
Oct 31 20:09:50.359330 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.359277    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:09:50.359396 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.359380    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:09:50.418384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.418344    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.418384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.418388    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mgs\" (UniqueName: \"kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.418637 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.418459    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.519716 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.519669    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.519716 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.519722    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mgs\" (UniqueName: \"kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.519941 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.519759    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.520512 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.520478    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.522006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.521985    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.533254 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.533235    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mgs\" (UniqueName: \"kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs\") pod \"maas-api-revoke-sub-c1e271a9-vh96n\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.665683 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.665651    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:50.944954 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:50.944920    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"]
Oct 31 20:09:50.946805 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:09:50.946779    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49804714_b3b3_4cee_8eec_b4bdf1abedee.slice/crio-012374937b9ccc4f76ddc545cfff6b96efb0da4c52747c1023a00466d6270483 WatchSource:0}: Error finding container 012374937b9ccc4f76ddc545cfff6b96efb0da4c52747c1023a00466d6270483: Status 404 returned error can't find the container with id 012374937b9ccc4f76ddc545cfff6b96efb0da4c52747c1023a00466d6270483
Oct 31 20:09:51.767641 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:51.767604    5958 generic.go:356] "Generic (PLEG): container finished" podID="49804714-b3b3-4cee-8eec-b4bdf1abedee" containerID="6cede83966f99e1078a892de536d9787e5ee1132e21797f08ea59673dd8fcd2d" exitCode=0
Oct 31 20:09:51.768092 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:51.767673    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n" event={"ID":"49804714-b3b3-4cee-8eec-b4bdf1abedee","Type":"ContainerDied","Data":"6cede83966f99e1078a892de536d9787e5ee1132e21797f08ea59673dd8fcd2d"}
Oct 31 20:09:51.768092 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:51.767733    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n" event={"ID":"49804714-b3b3-4cee-8eec-b4bdf1abedee","Type":"ContainerStarted","Data":"012374937b9ccc4f76ddc545cfff6b96efb0da4c52747c1023a00466d6270483"}
Oct 31 20:09:52.910230 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:52.910204    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:53.052771 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.052686    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca\") pod \"49804714-b3b3-4cee-8eec-b4bdf1abedee\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") "
Oct 31 20:09:53.052771 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.052754    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs\" (UniqueName: \"kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs\") pod \"49804714-b3b3-4cee-8eec-b4bdf1abedee\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") "
Oct 31 20:09:53.052977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.052784    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca\") pod \"49804714-b3b3-4cee-8eec-b4bdf1abedee\" (UID: \"49804714-b3b3-4cee-8eec-b4bdf1abedee\") "
Oct 31 20:09:53.053245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.053206    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca" pod "49804714-b3b3-4cee-8eec-b4bdf1abedee" (UID: "49804714-b3b3-4cee-8eec-b4bdf1abedee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:09:53.054868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.054843    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca" pod "49804714-b3b3-4cee-8eec-b4bdf1abedee" (UID: "49804714-b3b3-4cee-8eec-b4bdf1abedee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:09:53.054978 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.054956    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs" pod "49804714-b3b3-4cee-8eec-b4bdf1abedee" (UID: "49804714-b3b3-4cee-8eec-b4bdf1abedee"). InnerVolumeSpecName "kube-api-access-58mgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:53.154153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.154112    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:53.154153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.154146    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mgs\" (UniqueName: \"kubernetes.io/projected/49804714-b3b3-4cee-8eec-b4bdf1abedee-kube-api-access-58mgs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:53.154153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.154159    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49804714-b3b3-4cee-8eec-b4bdf1abedee-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:09:53.778255 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.778224    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"
Oct 31 20:09:53.778442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.778258    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n" event={"ID":"49804714-b3b3-4cee-8eec-b4bdf1abedee","Type":"ContainerDied","Data":"012374937b9ccc4f76ddc545cfff6b96efb0da4c52747c1023a00466d6270483"}
Oct 31 20:09:53.778442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:09:53.778305    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012374937b9ccc4f76ddc545cfff6b96efb0da4c52747c1023a00466d6270483"
Oct 31 20:10:01.309985 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:01.309954    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:10:06.633189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:06.633144    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:10:12.443175 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.443139    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"]
Oct 31 20:10:12.443726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.443704    5958 cpu_manager.go:414] "containerMap: removing container" podUID="49804714-b3b3-4cee-8eec-b4bdf1abedee" containerName="revoke-keys"
Oct 31 20:10:12.443726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.443720    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49804714-b3b3-4cee-8eec-b4bdf1abedee" containerName="revoke-keys"
Oct 31 20:10:12.443856 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.443820    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="49804714-b3b3-4cee-8eec-b4bdf1abedee" containerName="revoke-keys"
Oct 31 20:10:12.447421 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.447401    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.453126 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.453080    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:10:12.455439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.455386    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:10:12.458127 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.458104    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"]
Oct 31 20:10:12.561133 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.561090    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.561309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.561158    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvlv\" (UniqueName: \"kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.561309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.561250    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.662842 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.662807    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.663015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.662925    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.663015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.662958    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvlv\" (UniqueName: \"kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.663430 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.663405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.665180 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.665158    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.678737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.678714    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvlv\" (UniqueName: \"kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv\") pod \"maas-api-revoke-sub-2e1f5b5f-5qzp9\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:12.762794 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:12.762709    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:13.041202 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:13.041173    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"]
Oct 31 20:10:13.042901 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:10:13.042869    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode824a49f_a4f8_4ac0_aa2f_944341f7bf12.slice/crio-3e0f0f60468010b63b9165213bfcc6873be98e7a5fa05b2607d8eedd53646d54 WatchSource:0}: Error finding container 3e0f0f60468010b63b9165213bfcc6873be98e7a5fa05b2607d8eedd53646d54: Status 404 returned error can't find the container with id 3e0f0f60468010b63b9165213bfcc6873be98e7a5fa05b2607d8eedd53646d54
Oct 31 20:10:13.921514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:13.921476    5958 generic.go:356] "Generic (PLEG): container finished" podID="e824a49f-a4f8-4ac0-aa2f-944341f7bf12" containerID="42118ef7376c6add03b5bc8e4ad61a9e00369904b62ffa58f4853c99f2063ac2" exitCode=0
Oct 31 20:10:13.921952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:13.921527    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9" event={"ID":"e824a49f-a4f8-4ac0-aa2f-944341f7bf12","Type":"ContainerDied","Data":"42118ef7376c6add03b5bc8e4ad61a9e00369904b62ffa58f4853c99f2063ac2"}
Oct 31 20:10:13.921952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:13.921588    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9" event={"ID":"e824a49f-a4f8-4ac0-aa2f-944341f7bf12","Type":"ContainerStarted","Data":"3e0f0f60468010b63b9165213bfcc6873be98e7a5fa05b2607d8eedd53646d54"}
Oct 31 20:10:15.068384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.068359    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:15.190851 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.190761    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca\") pod \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") "
Oct 31 20:10:15.191016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.190877    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv\" (UniqueName: \"kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv\") pod \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") "
Oct 31 20:10:15.191016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.190910    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca\") pod \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\" (UID: \"e824a49f-a4f8-4ac0-aa2f-944341f7bf12\") "
Oct 31 20:10:15.191206 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.191182    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca" pod "e824a49f-a4f8-4ac0-aa2f-944341f7bf12" (UID: "e824a49f-a4f8-4ac0-aa2f-944341f7bf12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:15.191310 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.191288    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:15.193059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.193031    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv" pod "e824a49f-a4f8-4ac0-aa2f-944341f7bf12" (UID: "e824a49f-a4f8-4ac0-aa2f-944341f7bf12"). InnerVolumeSpecName "kube-api-access-5dvlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:15.193196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.193088    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca" pod "e824a49f-a4f8-4ac0-aa2f-944341f7bf12" (UID: "e824a49f-a4f8-4ac0-aa2f-944341f7bf12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:15.292033 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.291992    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvlv\" (UniqueName: \"kubernetes.io/projected/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-kube-api-access-5dvlv\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:15.292033 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.292034    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e824a49f-a4f8-4ac0-aa2f-944341f7bf12-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:15.934934 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.934906    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"
Oct 31 20:10:15.934934 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.934924    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9" event={"ID":"e824a49f-a4f8-4ac0-aa2f-944341f7bf12","Type":"ContainerDied","Data":"3e0f0f60468010b63b9165213bfcc6873be98e7a5fa05b2607d8eedd53646d54"}
Oct 31 20:10:15.935233 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:15.934967    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e0f0f60468010b63b9165213bfcc6873be98e7a5fa05b2607d8eedd53646d54"
Oct 31 20:10:17.043683 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:10:17.043634    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:10:23.321074 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.321034    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:10:23.456371 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.456337    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"]
Oct 31 20:10:23.456831 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.456816    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e824a49f-a4f8-4ac0-aa2f-944341f7bf12" containerName="revoke-keys"
Oct 31 20:10:23.456885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.456834    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e824a49f-a4f8-4ac0-aa2f-944341f7bf12" containerName="revoke-keys"
Oct 31 20:10:23.456960 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.456947    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e824a49f-a4f8-4ac0-aa2f-944341f7bf12" containerName="revoke-keys"
Oct 31 20:10:23.460798 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.460778    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.466812 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.466774    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:10:23.469505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.469482    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:10:23.469649 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.469617    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"]
Oct 31 20:10:23.576668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.576577    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.576668 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.576630    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd6d\" (UniqueName: \"kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.576877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.576745    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.677854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.677819    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.677854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.677858    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd6d\" (UniqueName: \"kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.678125 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.677972    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.678496 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.678475    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.680354 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.680327    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.691993 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.691967    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd6d\" (UniqueName: \"kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d\") pod \"maas-api-revoke-sub-ec843018-bg5w6\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:23.775948 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:23.775911    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:24.089106 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:24.089077    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"]
Oct 31 20:10:24.089820 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:10:24.089792    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b895bf5_89ca_40a2_927c_aee96b6bbcc7.slice/crio-13249fe9b1ab465ace3d9bd7257d97937f7443308c97f80c2790f8eda6ea96f7 WatchSource:0}: Error finding container 13249fe9b1ab465ace3d9bd7257d97937f7443308c97f80c2790f8eda6ea96f7: Status 404 returned error can't find the container with id 13249fe9b1ab465ace3d9bd7257d97937f7443308c97f80c2790f8eda6ea96f7
Oct 31 20:10:24.983596 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:24.983533    5958 generic.go:356] "Generic (PLEG): container finished" podID="8b895bf5-89ca-40a2-927c-aee96b6bbcc7" containerID="5918de8d41381ae1896a012c8a875f049b660ca13214624102754e7baa01c6ea" exitCode=0
Oct 31 20:10:24.983596 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:24.983582    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6" event={"ID":"8b895bf5-89ca-40a2-927c-aee96b6bbcc7","Type":"ContainerDied","Data":"5918de8d41381ae1896a012c8a875f049b660ca13214624102754e7baa01c6ea"}
Oct 31 20:10:24.984024 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:24.983637    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6" event={"ID":"8b895bf5-89ca-40a2-927c-aee96b6bbcc7","Type":"ContainerStarted","Data":"13249fe9b1ab465ace3d9bd7257d97937f7443308c97f80c2790f8eda6ea96f7"}
Oct 31 20:10:26.194423 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.194389    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:26.304526 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.304437    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d\" (UniqueName: \"kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d\") pod \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") "
Oct 31 20:10:26.304526 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.304491    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca\") pod \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") "
Oct 31 20:10:26.304800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.304635    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca\") pod \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\" (UID: \"8b895bf5-89ca-40a2-927c-aee96b6bbcc7\") "
Oct 31 20:10:26.304991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.304960    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca" pod "8b895bf5-89ca-40a2-927c-aee96b6bbcc7" (UID: "8b895bf5-89ca-40a2-927c-aee96b6bbcc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:26.306680 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.306655    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d" pod "8b895bf5-89ca-40a2-927c-aee96b6bbcc7" (UID: "8b895bf5-89ca-40a2-927c-aee96b6bbcc7"). InnerVolumeSpecName "kube-api-access-bmd6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:26.307248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.307221    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca" pod "8b895bf5-89ca-40a2-927c-aee96b6bbcc7" (UID: "8b895bf5-89ca-40a2-927c-aee96b6bbcc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:26.409514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.409464    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmd6d\" (UniqueName: \"kubernetes.io/projected/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-kube-api-access-bmd6d\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:26.409814 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.409790    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:26.409977 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:26.409960    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b895bf5-89ca-40a2-927c-aee96b6bbcc7-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:27.007620 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:27.005532    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6" event={"ID":"8b895bf5-89ca-40a2-927c-aee96b6bbcc7","Type":"ContainerDied","Data":"13249fe9b1ab465ace3d9bd7257d97937f7443308c97f80c2790f8eda6ea96f7"}
Oct 31 20:10:27.007620 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:27.005666    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13249fe9b1ab465ace3d9bd7257d97937f7443308c97f80c2790f8eda6ea96f7"
Oct 31 20:10:27.007620 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:27.005760    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"
Oct 31 20:10:36.206266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:36.206231    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:10:47.046656 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:10:47.046603    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:10:49.859933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.859897    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"]
Oct 31 20:10:49.860393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.860380    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8b895bf5-89ca-40a2-927c-aee96b6bbcc7" containerName="revoke-keys"
Oct 31 20:10:49.860439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.860394    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b895bf5-89ca-40a2-927c-aee96b6bbcc7" containerName="revoke-keys"
Oct 31 20:10:49.860479 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.860463    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b895bf5-89ca-40a2-927c-aee96b6bbcc7" containerName="revoke-keys"
Oct 31 20:10:49.863520 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.863503    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:49.872693 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.872665    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:10:49.873412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.872688    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"]
Oct 31 20:10:49.873581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.872694    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:10:49.907745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.907707    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:49.907928 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.907846    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkz8m\" (UniqueName: \"kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:49.907928 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:49.907912    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.009526 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.009481    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkz8m\" (UniqueName: \"kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.009771 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.009579    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.009771 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.009631    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.010249 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.010222    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.012029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.012007    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.024092 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.024057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkz8m\" (UniqueName: \"kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m\") pod \"maas-api-revoke-sub-046e1343-f95mg\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.181948 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.181907    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:50.461295 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:50.461269    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"]
Oct 31 20:10:50.463084 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:10:50.463048    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bf5a77_bdcf_468b_9ab2_c9e77d29a55b.slice/crio-120ca690fbb09b42043035dd68758ad389d6abfdf3ef1ba239d2f14aa311594a WatchSource:0}: Error finding container 120ca690fbb09b42043035dd68758ad389d6abfdf3ef1ba239d2f14aa311594a: Status 404 returned error can't find the container with id 120ca690fbb09b42043035dd68758ad389d6abfdf3ef1ba239d2f14aa311594a
Oct 31 20:10:51.155393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:51.155354    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg" event={"ID":"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b","Type":"ContainerStarted","Data":"2a916ac4723ce97d14a39e77d9deb31e20cd48f6f60b963248020748b83864be"}
Oct 31 20:10:51.155829 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:51.155410    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg" event={"ID":"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b","Type":"ContainerStarted","Data":"120ca690fbb09b42043035dd68758ad389d6abfdf3ef1ba239d2f14aa311594a"}
Oct 31 20:10:51.177481 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:51.177429    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg" podStartSLOduration=2.17741573 podStartE2EDuration="2.17741573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:10:51.176857729 +0000 UTC m=+3044.638232345" watchObservedRunningTime="2026-10-31 20:10:51.17741573 +0000 UTC m=+3044.638790341"
Oct 31 20:10:52.163011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:52.162972    5958 generic.go:356] "Generic (PLEG): container finished" podID="f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" containerID="2a916ac4723ce97d14a39e77d9deb31e20cd48f6f60b963248020748b83864be" exitCode=0
Oct 31 20:10:52.163453 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:52.163072    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg" event={"ID":"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b","Type":"ContainerDied","Data":"2a916ac4723ce97d14a39e77d9deb31e20cd48f6f60b963248020748b83864be"}
Oct 31 20:10:53.318211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.318181    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:53.446189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.446077    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca\") pod \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") "
Oct 31 20:10:53.446189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.446144    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m\" (UniqueName: \"kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m\") pod \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") "
Oct 31 20:10:53.446428 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.446243    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca\") pod \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\" (UID: \"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b\") "
Oct 31 20:10:53.446647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.446621    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca" pod "f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" (UID: "f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:53.446738 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.446719    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:53.448286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.448265    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca" pod "f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" (UID: "f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:53.448382 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.448365    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m" pod "f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" (UID: "f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b"). InnerVolumeSpecName "kube-api-access-fkz8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:53.548122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.548075    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:53.548122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:53.548118    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkz8m\" (UniqueName: \"kubernetes.io/projected/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b-kube-api-access-fkz8m\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:10:54.175807 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:54.175772    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"
Oct 31 20:10:54.175987 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:54.175800    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg" event={"ID":"f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b","Type":"ContainerDied","Data":"120ca690fbb09b42043035dd68758ad389d6abfdf3ef1ba239d2f14aa311594a"}
Oct 31 20:10:54.175987 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:10:54.175851    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120ca690fbb09b42043035dd68758ad389d6abfdf3ef1ba239d2f14aa311594a"
Oct 31 20:11:00.091436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.091395    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"]
Oct 31 20:11:00.091927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.091910    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" containerName="revoke-keys"
Oct 31 20:11:00.091973 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.091928    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" containerName="revoke-keys"
Oct 31 20:11:00.092028 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.092018    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" containerName="revoke-keys"
Oct 31 20:11:00.095393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.095376    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.105098 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.105070    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"]
Oct 31 20:11:00.130871 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.130832    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:00.131056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.130971    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:11:00.219069 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.219027    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.219248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.219088    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df88w\" (UniqueName: \"kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.219248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.219180    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.320561 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.320500    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.320725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.320613    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df88w\" (UniqueName: \"kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.320725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.320639    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.321374 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.321348    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.322918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.322893    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.334633 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.334604    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df88w\" (UniqueName: \"kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w\") pod \"maas-api-revoke-sub-d3fba056-5tjlh\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.415469 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.415420    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:00.693020 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.692986    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"]
Oct 31 20:11:00.694394 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:11:00.694364    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7463aa_758b_4109_b136_68e80dea7ec4.slice/crio-dc169ee9d2b2fd6e83323b8d514351196e87a1f3efc433b248395a0009395d2f WatchSource:0}: Error finding container dc169ee9d2b2fd6e83323b8d514351196e87a1f3efc433b248395a0009395d2f: Status 404 returned error can't find the container with id dc169ee9d2b2fd6e83323b8d514351196e87a1f3efc433b248395a0009395d2f
Oct 31 20:11:00.808269 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:00.808225    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:01.220201 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:01.220156    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh" event={"ID":"9e7463aa-758b-4109-b136-68e80dea7ec4","Type":"ContainerStarted","Data":"1ea2817ce34aeff6c381cd4c0512feb19f305aa3c4d617733ce912a99f5777e5"}
Oct 31 20:11:01.220644 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:01.220230    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh" event={"ID":"9e7463aa-758b-4109-b136-68e80dea7ec4","Type":"ContainerStarted","Data":"dc169ee9d2b2fd6e83323b8d514351196e87a1f3efc433b248395a0009395d2f"}
Oct 31 20:11:01.242322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:01.242269    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh" podStartSLOduration=1.242255471 podStartE2EDuration="1.242255471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:01.24195944 +0000 UTC m=+3054.703334055" watchObservedRunningTime="2026-10-31 20:11:01.242255471 +0000 UTC m=+3054.703630086"
Oct 31 20:11:02.260581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:02.259489    5958 generic.go:356] "Generic (PLEG): container finished" podID="9e7463aa-758b-4109-b136-68e80dea7ec4" containerID="1ea2817ce34aeff6c381cd4c0512feb19f305aa3c4d617733ce912a99f5777e5" exitCode=0
Oct 31 20:11:02.260581 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:02.259670    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh" event={"ID":"9e7463aa-758b-4109-b136-68e80dea7ec4","Type":"ContainerDied","Data":"1ea2817ce34aeff6c381cd4c0512feb19f305aa3c4d617733ce912a99f5777e5"}
Oct 31 20:11:03.612889 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.612367    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:03.793995 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.793951    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca\") pod \"9e7463aa-758b-4109-b136-68e80dea7ec4\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") "
Oct 31 20:11:03.794212 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.794027    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w\" (UniqueName: \"kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w\") pod \"9e7463aa-758b-4109-b136-68e80dea7ec4\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") "
Oct 31 20:11:03.794212 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.794100    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca\") pod \"9e7463aa-758b-4109-b136-68e80dea7ec4\" (UID: \"9e7463aa-758b-4109-b136-68e80dea7ec4\") "
Oct 31 20:11:03.794475 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.794447    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca" pod "9e7463aa-758b-4109-b136-68e80dea7ec4" (UID: "9e7463aa-758b-4109-b136-68e80dea7ec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:03.796461 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.796421    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca" pod "9e7463aa-758b-4109-b136-68e80dea7ec4" (UID: "9e7463aa-758b-4109-b136-68e80dea7ec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:03.796631 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.796567    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w" pod "9e7463aa-758b-4109-b136-68e80dea7ec4" (UID: "9e7463aa-758b-4109-b136-68e80dea7ec4"). InnerVolumeSpecName "kube-api-access-df88w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:03.895746 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.895712    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df88w\" (UniqueName: \"kubernetes.io/projected/9e7463aa-758b-4109-b136-68e80dea7ec4-kube-api-access-df88w\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:03.895746 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.895745    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:03.895988 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:03.895761    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e7463aa-758b-4109-b136-68e80dea7ec4-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:04.282933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:04.282857    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"
Oct 31 20:11:04.282933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:04.282891    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh" event={"ID":"9e7463aa-758b-4109-b136-68e80dea7ec4","Type":"ContainerDied","Data":"dc169ee9d2b2fd6e83323b8d514351196e87a1f3efc433b248395a0009395d2f"}
Oct 31 20:11:04.282933 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:04.282936    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc169ee9d2b2fd6e83323b8d514351196e87a1f3efc433b248395a0009395d2f"
Oct 31 20:11:11.087908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:11.087874    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:16.238256 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:16.238157    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:17.045217 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:11:17.045171    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:11:21.125169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.125131    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"]
Oct 31 20:11:21.125714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.125600    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9e7463aa-758b-4109-b136-68e80dea7ec4" containerName="revoke-keys"
Oct 31 20:11:21.125714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.125612    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7463aa-758b-4109-b136-68e80dea7ec4" containerName="revoke-keys"
Oct 31 20:11:21.125793 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.125720    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7463aa-758b-4109-b136-68e80dea7ec4" containerName="revoke-keys"
Oct 31 20:11:21.129033 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.129016    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.138678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.138636    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:21.138834 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.138702    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:11:21.140764 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.140744    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"]
Oct 31 20:11:21.300987 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.300944    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtdc\" (UniqueName: \"kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.301177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.301005    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.301177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.301085    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.402384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.402297    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtdc\" (UniqueName: \"kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.402384 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.402337    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.402612 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.402458    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.403008 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.402989    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.404970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.404943    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.421702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.421673    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtdc\" (UniqueName: \"kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc\") pod \"maas-api-revoke-sub-ae594979-wjczb\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.443497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.443471    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:21.738122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:21.738089    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"]
Oct 31 20:11:21.740083 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:11:21.740055    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fe52a0_3238_48a5_bcdb_b8ac1ed79f21.slice/crio-45779b4d7091970c8a52659e9b1763158413116f6ed8ae42482069f844ef3393 WatchSource:0}: Error finding container 45779b4d7091970c8a52659e9b1763158413116f6ed8ae42482069f844ef3393: Status 404 returned error can't find the container with id 45779b4d7091970c8a52659e9b1763158413116f6ed8ae42482069f844ef3393
Oct 31 20:11:22.408537 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:22.408501    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb" event={"ID":"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21","Type":"ContainerStarted","Data":"bc3ccf7bd9bbe1f510bb6eca672988ccce8fb6eca7729e4842a6fff02d2df707"}
Oct 31 20:11:22.408945 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:22.408568    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb" event={"ID":"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21","Type":"ContainerStarted","Data":"45779b4d7091970c8a52659e9b1763158413116f6ed8ae42482069f844ef3393"}
Oct 31 20:11:22.432385 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:22.432331    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb" podStartSLOduration=1.432317868 podStartE2EDuration="1.432317868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:22.42910434 +0000 UTC m=+3075.890478955" watchObservedRunningTime="2026-10-31 20:11:22.432317868 +0000 UTC m=+3075.893692482"
Oct 31 20:11:23.414680 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:23.414647    5958 generic.go:356] "Generic (PLEG): container finished" podID="23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" containerID="bc3ccf7bd9bbe1f510bb6eca672988ccce8fb6eca7729e4842a6fff02d2df707" exitCode=0
Oct 31 20:11:23.415154 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:23.414695    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb" event={"ID":"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21","Type":"ContainerDied","Data":"bc3ccf7bd9bbe1f510bb6eca672988ccce8fb6eca7729e4842a6fff02d2df707"}
Oct 31 20:11:24.565635 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.565601    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:24.739590 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.739483    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca\") pod \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") "
Oct 31 20:11:24.739749 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.739667    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca\") pod \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") "
Oct 31 20:11:24.739811 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.739793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc\" (UniqueName: \"kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc\") pod \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\" (UID: \"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21\") "
Oct 31 20:11:24.739895 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.739866    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca" pod "23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" (UID: "23fe52a0-3238-48a5-bcdb-b8ac1ed79f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:24.740188 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.740164    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:24.741691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.741672    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca" pod "23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" (UID: "23fe52a0-3238-48a5-bcdb-b8ac1ed79f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:24.741808 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.741769    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc" pod "23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" (UID: "23fe52a0-3238-48a5-bcdb-b8ac1ed79f21"). InnerVolumeSpecName "kube-api-access-dgtdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:24.841458 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.841422    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:24.841458 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:24.841454    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtdc\" (UniqueName: \"kubernetes.io/projected/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21-kube-api-access-dgtdc\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:25.427144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:25.427108    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb" event={"ID":"23fe52a0-3238-48a5-bcdb-b8ac1ed79f21","Type":"ContainerDied","Data":"45779b4d7091970c8a52659e9b1763158413116f6ed8ae42482069f844ef3393"}
Oct 31 20:11:25.427426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:25.427148    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"
Oct 31 20:11:25.427426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:25.427158    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45779b4d7091970c8a52659e9b1763158413116f6ed8ae42482069f844ef3393"
Oct 31 20:11:32.095315 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:32.095276    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:35.768219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:35.768148    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:39.398146 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.398102    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"]
Oct 31 20:11:39.398678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.398620    5958 cpu_manager.go:414] "containerMap: removing container" podUID="23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" containerName="revoke-keys"
Oct 31 20:11:39.398678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.398636    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" containerName="revoke-keys"
Oct 31 20:11:39.398809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.398720    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" containerName="revoke-keys"
Oct 31 20:11:39.402191 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.402169    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.410149 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.410112    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:39.410445 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.410244    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:11:39.411020 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.410994    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"]
Oct 31 20:11:39.423471 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.423431    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.423745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.423648    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.423852 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.423756    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95h4\" (UniqueName: \"kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.525145 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.525105    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95h4\" (UniqueName: \"kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.525342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.525196    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.525342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.525314    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.525991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.525969    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.527765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.527741    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.538809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.538781    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95h4\" (UniqueName: \"kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4\") pod \"maas-api-revoke-sub-a4aef7d7-7fhzj\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.718689 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.718595    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:39.999049 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:39.999015    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"]
Oct 31 20:11:40.515475 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:40.515445    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj" event={"ID":"7a841f05-b630-4be5-9a05-6f60bab75caa","Type":"ContainerStarted","Data":"46f303e1740c36b88d614d0dbe3f63908277023f26bb8b9d03561d7f1cb8b996"}
Oct 31 20:11:40.515892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:40.515497    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj" event={"ID":"7a841f05-b630-4be5-9a05-6f60bab75caa","Type":"ContainerStarted","Data":"e7755a7c8dca0322fd5f65dcf504d08e96271d64332002d7cf19ceceac151bd8"}
Oct 31 20:11:40.538107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:40.538055    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj" podStartSLOduration=1.53804147 podStartE2EDuration="1.53804147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:40.535807658 +0000 UTC m=+3093.997182276" watchObservedRunningTime="2026-10-31 20:11:40.53804147 +0000 UTC m=+3093.999416077"
Oct 31 20:11:41.523122 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:41.523083    5958 generic.go:356] "Generic (PLEG): container finished" podID="7a841f05-b630-4be5-9a05-6f60bab75caa" containerID="46f303e1740c36b88d614d0dbe3f63908277023f26bb8b9d03561d7f1cb8b996" exitCode=0
Oct 31 20:11:41.523570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:41.523167    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj" event={"ID":"7a841f05-b630-4be5-9a05-6f60bab75caa","Type":"ContainerDied","Data":"46f303e1740c36b88d614d0dbe3f63908277023f26bb8b9d03561d7f1cb8b996"}
Oct 31 20:11:42.674946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.674919    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:42.760836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.760800    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca\") pod \"7a841f05-b630-4be5-9a05-6f60bab75caa\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") "
Oct 31 20:11:42.761028 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.760952    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca\") pod \"7a841f05-b630-4be5-9a05-6f60bab75caa\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") "
Oct 31 20:11:42.761028 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.760981    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4\" (UniqueName: \"kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4\") pod \"7a841f05-b630-4be5-9a05-6f60bab75caa\" (UID: \"7a841f05-b630-4be5-9a05-6f60bab75caa\") "
Oct 31 20:11:42.761266 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.761235    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca" pod "7a841f05-b630-4be5-9a05-6f60bab75caa" (UID: "7a841f05-b630-4be5-9a05-6f60bab75caa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:42.761405 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.761347    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:42.763178 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.763153    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4" pod "7a841f05-b630-4be5-9a05-6f60bab75caa" (UID: "7a841f05-b630-4be5-9a05-6f60bab75caa"). InnerVolumeSpecName "kube-api-access-l95h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:42.763286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.763154    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca" pod "7a841f05-b630-4be5-9a05-6f60bab75caa" (UID: "7a841f05-b630-4be5-9a05-6f60bab75caa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:42.862160 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.862069    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a841f05-b630-4be5-9a05-6f60bab75caa-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:42.862160 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:42.862104    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95h4\" (UniqueName: \"kubernetes.io/projected/7a841f05-b630-4be5-9a05-6f60bab75caa-kube-api-access-l95h4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:43.537093 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:43.537051    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"
Oct 31 20:11:43.537365 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:43.537054    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj" event={"ID":"7a841f05-b630-4be5-9a05-6f60bab75caa","Type":"ContainerDied","Data":"e7755a7c8dca0322fd5f65dcf504d08e96271d64332002d7cf19ceceac151bd8"}
Oct 31 20:11:43.537365 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:43.537191    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7755a7c8dca0322fd5f65dcf504d08e96271d64332002d7cf19ceceac151bd8"
Oct 31 20:11:47.044468 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:11:47.044421    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:11:50.281986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:50.281942    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:55.498522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:55.498483    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:11:58.348940 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.348900    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"]
Oct 31 20:11:58.349583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.349531    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7a841f05-b630-4be5-9a05-6f60bab75caa" containerName="revoke-keys"
Oct 31 20:11:58.349583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.349571    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a841f05-b630-4be5-9a05-6f60bab75caa" containerName="revoke-keys"
Oct 31 20:11:58.349792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.349734    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a841f05-b630-4be5-9a05-6f60bab75caa" containerName="revoke-keys"
Oct 31 20:11:58.354341 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.354315    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.362302 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.361567    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:11:58.362302 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.361626    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:11:58.362302 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.362228    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"]
Oct 31 20:11:58.430221 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.430179    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.430412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.430263    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.430412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.430330    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7qn\" (UniqueName: \"kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.531120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.531073    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.531329 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.531153    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.531329 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.531175    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7qn\" (UniqueName: \"kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.531919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.531892    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.533375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.533355    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.550784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.550756    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7qn\" (UniqueName: \"kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn\") pod \"maas-api-revoke-sub-79805ca9-f5tgw\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.671457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.671429    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:11:58.946870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:58.946836    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"]
Oct 31 20:11:58.947951 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:11:58.947915    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb828b17c_2771_45fc_9f70_56c71c0133d5.slice/crio-309f61a275bc922c34a950fa304e7f31e359193cedcc2e71ae8d86d57f3ecaac WatchSource:0}: Error finding container 309f61a275bc922c34a950fa304e7f31e359193cedcc2e71ae8d86d57f3ecaac: Status 404 returned error can't find the container with id 309f61a275bc922c34a950fa304e7f31e359193cedcc2e71ae8d86d57f3ecaac
Oct 31 20:11:59.627986 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:59.627945    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw" event={"ID":"b828b17c-2771-45fc-9f70-56c71c0133d5","Type":"ContainerStarted","Data":"67760fe95d6c85a46ed562b36e58ac6f26e9551454af1f191c5ad6ccfa4adfb5"}
Oct 31 20:11:59.628395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:59.628013    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw" event={"ID":"b828b17c-2771-45fc-9f70-56c71c0133d5","Type":"ContainerStarted","Data":"309f61a275bc922c34a950fa304e7f31e359193cedcc2e71ae8d86d57f3ecaac"}
Oct 31 20:11:59.651197 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:11:59.651140    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw" podStartSLOduration=1.651126643 podStartE2EDuration="1.651126643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:11:59.649490717 +0000 UTC m=+3113.110865333" watchObservedRunningTime="2026-10-31 20:11:59.651126643 +0000 UTC m=+3113.112501284"
Oct 31 20:12:00.634905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:00.634870    5958 generic.go:356] "Generic (PLEG): container finished" podID="b828b17c-2771-45fc-9f70-56c71c0133d5" containerID="67760fe95d6c85a46ed562b36e58ac6f26e9551454af1f191c5ad6ccfa4adfb5" exitCode=0
Oct 31 20:12:00.635312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:00.634952    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw" event={"ID":"b828b17c-2771-45fc-9f70-56c71c0133d5","Type":"ContainerDied","Data":"67760fe95d6c85a46ed562b36e58ac6f26e9551454af1f191c5ad6ccfa4adfb5"}
Oct 31 20:12:01.790867 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.790840    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:12:01.865499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.865467    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn\" (UniqueName: \"kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn\") pod \"b828b17c-2771-45fc-9f70-56c71c0133d5\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") "
Oct 31 20:12:01.865747 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.865524    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca\") pod \"b828b17c-2771-45fc-9f70-56c71c0133d5\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") "
Oct 31 20:12:01.865747 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.865631    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca\") pod \"b828b17c-2771-45fc-9f70-56c71c0133d5\" (UID: \"b828b17c-2771-45fc-9f70-56c71c0133d5\") "
Oct 31 20:12:01.865959 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.865923    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca" pod "b828b17c-2771-45fc-9f70-56c71c0133d5" (UID: "b828b17c-2771-45fc-9f70-56c71c0133d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:01.867761 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.867725    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca" pod "b828b17c-2771-45fc-9f70-56c71c0133d5" (UID: "b828b17c-2771-45fc-9f70-56c71c0133d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:01.867885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.867799    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn" pod "b828b17c-2771-45fc-9f70-56c71c0133d5" (UID: "b828b17c-2771-45fc-9f70-56c71c0133d5"). InnerVolumeSpecName "kube-api-access-mh7qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:01.967234 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.967134    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:01.967234 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.967171    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7qn\" (UniqueName: \"kubernetes.io/projected/b828b17c-2771-45fc-9f70-56c71c0133d5-kube-api-access-mh7qn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:01.967234 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:01.967188    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828b17c-2771-45fc-9f70-56c71c0133d5-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:02.650699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:02.650672    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"
Oct 31 20:12:02.650892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:02.650675    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw" event={"ID":"b828b17c-2771-45fc-9f70-56c71c0133d5","Type":"ContainerDied","Data":"309f61a275bc922c34a950fa304e7f31e359193cedcc2e71ae8d86d57f3ecaac"}
Oct 31 20:12:02.650892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:02.650794    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309f61a275bc922c34a950fa304e7f31e359193cedcc2e71ae8d86d57f3ecaac"
Oct 31 20:12:09.209495 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:09.209458    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:12:12.977416 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:12.977365    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:12:16.166709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.166672    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"]
Oct 31 20:12:16.167157 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.167144    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b828b17c-2771-45fc-9f70-56c71c0133d5" containerName="revoke-keys"
Oct 31 20:12:16.167211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.167159    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b828b17c-2771-45fc-9f70-56c71c0133d5" containerName="revoke-keys"
Oct 31 20:12:16.167258 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.167247    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b828b17c-2771-45fc-9f70-56c71c0133d5" containerName="revoke-keys"
Oct 31 20:12:16.170753 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.170735    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.180169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.180140    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:12:16.180496 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.180468    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:12:16.180914 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.180884    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"]
Oct 31 20:12:16.219497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.219455    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.219730 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.219509    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtgx\" (UniqueName: \"kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.219730 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.219690    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.321236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.321200    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.321415 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.321248    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtgx\" (UniqueName: \"kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.321415 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.321302    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.321993 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.321974    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.323649 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.323630    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.335174 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.335150    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtgx\" (UniqueName: \"kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx\") pod \"maas-api-revoke-sub-0b7e8edb-gdl8m\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.487376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.487290    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:16.764460 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:16.764430    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"]
Oct 31 20:12:16.766019 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:12:16.765994    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac0f07e_da27_4fef_912e_93b448679650.slice/crio-b4570d08e51a8dfafed9e593fb7c4b89f07331cb8ba428008ef5cbef91e301e8 WatchSource:0}: Error finding container b4570d08e51a8dfafed9e593fb7c4b89f07331cb8ba428008ef5cbef91e301e8: Status 404 returned error can't find the container with id b4570d08e51a8dfafed9e593fb7c4b89f07331cb8ba428008ef5cbef91e301e8
Oct 31 20:12:17.045476 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:12:17.045438    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:12:17.768431 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:17.768333    5958 generic.go:356] "Generic (PLEG): container finished" podID="2ac0f07e-da27-4fef-912e-93b448679650" containerID="6a4ae94cd788d0401ca27647a7b06edd39fd89b69cca6b2474de6f2414dc2c47" exitCode=0
Oct 31 20:12:17.768431 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:17.768374    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m" event={"ID":"2ac0f07e-da27-4fef-912e-93b448679650","Type":"ContainerDied","Data":"6a4ae94cd788d0401ca27647a7b06edd39fd89b69cca6b2474de6f2414dc2c47"}
Oct 31 20:12:17.768945 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:17.768435    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m" event={"ID":"2ac0f07e-da27-4fef-912e-93b448679650","Type":"ContainerStarted","Data":"b4570d08e51a8dfafed9e593fb7c4b89f07331cb8ba428008ef5cbef91e301e8"}
Oct 31 20:12:18.915169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:18.915140    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:19.047784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.047691    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx\" (UniqueName: \"kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx\") pod \"2ac0f07e-da27-4fef-912e-93b448679650\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") "
Oct 31 20:12:19.047784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.047747    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca\") pod \"2ac0f07e-da27-4fef-912e-93b448679650\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") "
Oct 31 20:12:19.047980 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.047844    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca\") pod \"2ac0f07e-da27-4fef-912e-93b448679650\" (UID: \"2ac0f07e-da27-4fef-912e-93b448679650\") "
Oct 31 20:12:19.048212 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.048184    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca" pod "2ac0f07e-da27-4fef-912e-93b448679650" (UID: "2ac0f07e-da27-4fef-912e-93b448679650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:19.049883 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.049864    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca" pod "2ac0f07e-da27-4fef-912e-93b448679650" (UID: "2ac0f07e-da27-4fef-912e-93b448679650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:19.050003 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.049979    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx" pod "2ac0f07e-da27-4fef-912e-93b448679650" (UID: "2ac0f07e-da27-4fef-912e-93b448679650"). InnerVolumeSpecName "kube-api-access-8gtgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:19.148994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.148954    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:19.148994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.148986    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtgx\" (UniqueName: \"kubernetes.io/projected/2ac0f07e-da27-4fef-912e-93b448679650-kube-api-access-8gtgx\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:19.148994 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.148997    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac0f07e-da27-4fef-912e-93b448679650-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:19.780807 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.780776    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"
Oct 31 20:12:19.780995 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.780820    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m" event={"ID":"2ac0f07e-da27-4fef-912e-93b448679650","Type":"ContainerDied","Data":"b4570d08e51a8dfafed9e593fb7c4b89f07331cb8ba428008ef5cbef91e301e8"}
Oct 31 20:12:19.780995 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:19.780870    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4570d08e51a8dfafed9e593fb7c4b89f07331cb8ba428008ef5cbef91e301e8"
Oct 31 20:12:26.881450 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.881417    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"]
Oct 31 20:12:26.882006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.881938    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2ac0f07e-da27-4fef-912e-93b448679650" containerName="revoke-keys"
Oct 31 20:12:26.882006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.881952    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac0f07e-da27-4fef-912e-93b448679650" containerName="revoke-keys"
Oct 31 20:12:26.882090 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.882048    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac0f07e-da27-4fef-912e-93b448679650" containerName="revoke-keys"
Oct 31 20:12:26.885188 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.885160    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:26.895638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.894522    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:12:26.895638 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.894661    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:12:26.899910 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:26.899874    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"]
Oct 31 20:12:27.004692 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.004658    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:12:27.028560 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.028515    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.028748 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.028705    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzxc\" (UniqueName: \"kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.028843 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.028823    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.130481 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.130437    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzxc\" (UniqueName: \"kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.130729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.130586    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.130803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.130773    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.131378 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.131345    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.133040 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.132996    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.144392 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.144366    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzxc\" (UniqueName: \"kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc\") pod \"maas-api-revoke-sub-3d4156f1-5vzw5\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.203253 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.203208    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:27.488057 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.488008    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"]
Oct 31 20:12:27.491117 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:12:27.491076    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc7319b_a428_46ae_ab3e_6b8eec156431.slice/crio-668eb2eccf7945bf76306421dfb44227c6c9fe3a9e412eff0b8a1e2b13d60e92 WatchSource:0}: Error finding container 668eb2eccf7945bf76306421dfb44227c6c9fe3a9e412eff0b8a1e2b13d60e92: Status 404 returned error can't find the container with id 668eb2eccf7945bf76306421dfb44227c6c9fe3a9e412eff0b8a1e2b13d60e92
Oct 31 20:12:27.826916 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.826884    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5" event={"ID":"2bc7319b-a428-46ae-ab3e-6b8eec156431","Type":"ContainerStarted","Data":"239e3a3c8de5d7e0e9b4e989cff33e1f17c9db2b5fd4d344510fe73b4fe9ac3b"}
Oct 31 20:12:27.827082 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.826932    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5" event={"ID":"2bc7319b-a428-46ae-ab3e-6b8eec156431","Type":"ContainerStarted","Data":"668eb2eccf7945bf76306421dfb44227c6c9fe3a9e412eff0b8a1e2b13d60e92"}
Oct 31 20:12:27.848118 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:27.848069    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5" podStartSLOduration=1.848054838 podStartE2EDuration="1.848054838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:27.846638404 +0000 UTC m=+3141.308013019" watchObservedRunningTime="2026-10-31 20:12:27.848054838 +0000 UTC m=+3141.309429453"
Oct 31 20:12:28.833882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:28.833849    5958 generic.go:356] "Generic (PLEG): container finished" podID="2bc7319b-a428-46ae-ab3e-6b8eec156431" containerID="239e3a3c8de5d7e0e9b4e989cff33e1f17c9db2b5fd4d344510fe73b4fe9ac3b" exitCode=0
Oct 31 20:12:28.834381 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:28.833928    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5" event={"ID":"2bc7319b-a428-46ae-ab3e-6b8eec156431","Type":"ContainerDied","Data":"239e3a3c8de5d7e0e9b4e989cff33e1f17c9db2b5fd4d344510fe73b4fe9ac3b"}
Oct 31 20:12:29.984435 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:29.984408    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:30.072198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.072160    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc\" (UniqueName: \"kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc\") pod \"2bc7319b-a428-46ae-ab3e-6b8eec156431\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") "
Oct 31 20:12:30.072399 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.072318    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca\") pod \"2bc7319b-a428-46ae-ab3e-6b8eec156431\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") "
Oct 31 20:12:30.072399 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.072346    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca\") pod \"2bc7319b-a428-46ae-ab3e-6b8eec156431\" (UID: \"2bc7319b-a428-46ae-ab3e-6b8eec156431\") "
Oct 31 20:12:30.072791 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.072764    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca" pod "2bc7319b-a428-46ae-ab3e-6b8eec156431" (UID: "2bc7319b-a428-46ae-ab3e-6b8eec156431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:30.074332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.074297    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc" pod "2bc7319b-a428-46ae-ab3e-6b8eec156431" (UID: "2bc7319b-a428-46ae-ab3e-6b8eec156431"). InnerVolumeSpecName "kube-api-access-6kzxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:30.074332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.074300    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca" pod "2bc7319b-a428-46ae-ab3e-6b8eec156431" (UID: "2bc7319b-a428-46ae-ab3e-6b8eec156431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:30.174094 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.174056    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:30.174094 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.174091    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc7319b-a428-46ae-ab3e-6b8eec156431-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:30.174094 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.174103    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzxc\" (UniqueName: \"kubernetes.io/projected/2bc7319b-a428-46ae-ab3e-6b8eec156431-kube-api-access-6kzxc\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:30.846238 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.846208    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"
Oct 31 20:12:30.846433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.846240    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5" event={"ID":"2bc7319b-a428-46ae-ab3e-6b8eec156431","Type":"ContainerDied","Data":"668eb2eccf7945bf76306421dfb44227c6c9fe3a9e412eff0b8a1e2b13d60e92"}
Oct 31 20:12:30.846433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:30.846290    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="668eb2eccf7945bf76306421dfb44227c6c9fe3a9e412eff0b8a1e2b13d60e92"
Oct 31 20:12:37.812805 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:37.812769    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:12:41.878828 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:41.878796    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:12:45.032109 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.032070    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"]
Oct 31 20:12:45.037360 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.037330    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5vzw5"]
Oct 31 20:12:45.056064 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.056023    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"]
Oct 31 20:12:45.056605 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.056584    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2bc7319b-a428-46ae-ab3e-6b8eec156431" containerName="revoke-keys"
Oct 31 20:12:45.056605 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.056606    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc7319b-a428-46ae-ab3e-6b8eec156431" containerName="revoke-keys"
Oct 31 20:12:45.056772 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.056724    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc7319b-a428-46ae-ab3e-6b8eec156431" containerName="revoke-keys"
Oct 31 20:12:45.059972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.059951    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.069984 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.069956    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:12:45.072715 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.072671    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:12:45.076795 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.076768    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"]
Oct 31 20:12:45.137109 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.137057    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.137289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.137197    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncpb\" (UniqueName: \"kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.137289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.137270    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.238181 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.238142    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.238382 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.238291    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncpb\" (UniqueName: \"kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.238382 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.238360    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.238922 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.238892    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.240740 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.240711    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.252306 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.252276    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncpb\" (UniqueName: \"kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb\") pod \"maas-api-revoke-sub-3d4156f1-x6wpz\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.375714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.375684    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:45.656082 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.656055    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"]
Oct 31 20:12:45.658572 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:12:45.658522    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e76b29_cec5_465e_8da7_3f8e6323cd69.slice/crio-a5ed68f3f9a3290e6fe78704aa255279351b75886391c37885f72d92326ca0a6 WatchSource:0}: Error finding container a5ed68f3f9a3290e6fe78704aa255279351b75886391c37885f72d92326ca0a6: Status 404 returned error can't find the container with id a5ed68f3f9a3290e6fe78704aa255279351b75886391c37885f72d92326ca0a6
Oct 31 20:12:45.931860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.931816    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz" event={"ID":"28e76b29-cec5-465e-8da7-3f8e6323cd69","Type":"ContainerStarted","Data":"56234798b70ca104e412629332d572da6a31bce4be7e126ca67b0989574f6e7b"}
Oct 31 20:12:45.932049 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.931911    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz" event={"ID":"28e76b29-cec5-465e-8da7-3f8e6323cd69","Type":"ContainerStarted","Data":"a5ed68f3f9a3290e6fe78704aa255279351b75886391c37885f72d92326ca0a6"}
Oct 31 20:12:45.952843 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:45.952782    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz" podStartSLOduration=0.952768279 podStartE2EDuration="952.768279ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:12:45.952104647 +0000 UTC m=+3159.413479266" watchObservedRunningTime="2026-10-31 20:12:45.952768279 +0000 UTC m=+3159.414142902"
Oct 31 20:12:46.662477 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:46.662443    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc7319b-a428-46ae-ab3e-6b8eec156431" path="/var/lib/kubelet/pods/2bc7319b-a428-46ae-ab3e-6b8eec156431/volumes"
Oct 31 20:12:46.940301 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:46.940267    5958 generic.go:356] "Generic (PLEG): container finished" podID="28e76b29-cec5-465e-8da7-3f8e6323cd69" containerID="56234798b70ca104e412629332d572da6a31bce4be7e126ca67b0989574f6e7b" exitCode=0
Oct 31 20:12:46.940491 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:46.940310    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz" event={"ID":"28e76b29-cec5-465e-8da7-3f8e6323cd69","Type":"ContainerDied","Data":"56234798b70ca104e412629332d572da6a31bce4be7e126ca67b0989574f6e7b"}
Oct 31 20:12:47.048247 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:12:47.048206    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:12:48.093777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.093752    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:48.168130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.168097    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca\") pod \"28e76b29-cec5-465e-8da7-3f8e6323cd69\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") "
Oct 31 20:12:48.168588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.168161    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb\" (UniqueName: \"kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb\") pod \"28e76b29-cec5-465e-8da7-3f8e6323cd69\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") "
Oct 31 20:12:48.168588 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.168222    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca\") pod \"28e76b29-cec5-465e-8da7-3f8e6323cd69\" (UID: \"28e76b29-cec5-465e-8da7-3f8e6323cd69\") "
Oct 31 20:12:48.168697 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.168647    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca" pod "28e76b29-cec5-465e-8da7-3f8e6323cd69" (UID: "28e76b29-cec5-465e-8da7-3f8e6323cd69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:12:48.170288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.170262    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca" pod "28e76b29-cec5-465e-8da7-3f8e6323cd69" (UID: "28e76b29-cec5-465e-8da7-3f8e6323cd69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:12:48.170288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.170271    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb" pod "28e76b29-cec5-465e-8da7-3f8e6323cd69" (UID: "28e76b29-cec5-465e-8da7-3f8e6323cd69"). InnerVolumeSpecName "kube-api-access-5ncpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:12:48.269497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.269416    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ncpb\" (UniqueName: \"kubernetes.io/projected/28e76b29-cec5-465e-8da7-3f8e6323cd69-kube-api-access-5ncpb\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:48.269497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.269446    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:48.269497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.269457    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e76b29-cec5-465e-8da7-3f8e6323cd69-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:12:48.953367 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.953281    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz" event={"ID":"28e76b29-cec5-465e-8da7-3f8e6323cd69","Type":"ContainerDied","Data":"a5ed68f3f9a3290e6fe78704aa255279351b75886391c37885f72d92326ca0a6"}
Oct 31 20:12:48.953367 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.953332    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ed68f3f9a3290e6fe78704aa255279351b75886391c37885f72d92326ca0a6"
Oct 31 20:12:48.953367 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:48.953296    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"
Oct 31 20:12:56.517763 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:12:56.517723    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:13:00.612235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:00.612196    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:13:06.310322 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:06.310281    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:13:12.036945 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:12.036907    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"]
Oct 31 20:13:12.039462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:12.039439    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1e607a-nqmfk"]
Oct 31 20:13:12.663079 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:12.663038    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcef7e2b-796e-4dc4-83dd-a6c50a91d894" path="/var/lib/kubelet/pods/fcef7e2b-796e-4dc4-83dd-a6c50a91d894/volumes"
Oct 31 20:13:13.874153 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.874115    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"]
Oct 31 20:13:13.874657 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.874642    5958 cpu_manager.go:414] "containerMap: removing container" podUID="28e76b29-cec5-465e-8da7-3f8e6323cd69" containerName="revoke-keys"
Oct 31 20:13:13.874731 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.874658    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e76b29-cec5-465e-8da7-3f8e6323cd69" containerName="revoke-keys"
Oct 31 20:13:13.874779 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.874768    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e76b29-cec5-465e-8da7-3f8e6323cd69" containerName="revoke-keys"
Oct 31 20:13:13.878524 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.878503    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:13.891174 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.891146    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"]
Oct 31 20:13:13.892164 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.892141    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:13:13.892279 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.892178    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:13:13.937042 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.936999    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:13.937256 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.937161    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:13.937256 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:13.937196    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.038725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.038683    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.038947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.038782    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.038947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.038805    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.039433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.039412    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.041196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.041171    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.052349 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.052321    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw\") pod \"maas-api-revoke-sub-e2fc9c5b-d6l9h\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.193709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.193608    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:14.473853 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:14.473829    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"]
Oct 31 20:13:14.475226 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:13:14.475199    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051ded6c_6f9f_4df6_bb2d_c4b4dc803c40.slice/crio-8de3d934474025ed04081a2c201f9f4b4bbe5c0a85c4bc72986dc781acc5c8d1 WatchSource:0}: Error finding container 8de3d934474025ed04081a2c201f9f4b4bbe5c0a85c4bc72986dc781acc5c8d1: Status 404 returned error can't find the container with id 8de3d934474025ed04081a2c201f9f4b4bbe5c0a85c4bc72986dc781acc5c8d1
Oct 31 20:13:15.124437 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:15.124405    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h" event={"ID":"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40","Type":"ContainerStarted","Data":"a519fa1c428d4b03c340f72b680f70dbfccc7c3f2d94f021ba4b9858e935838c"}
Oct 31 20:13:15.124858 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:15.124458    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h" event={"ID":"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40","Type":"ContainerStarted","Data":"8de3d934474025ed04081a2c201f9f4b4bbe5c0a85c4bc72986dc781acc5c8d1"}
Oct 31 20:13:15.148733 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:15.148677    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h" podStartSLOduration=2.14866393 podStartE2EDuration="2.14866393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:13:15.147143115 +0000 UTC m=+3188.608517727" watchObservedRunningTime="2026-10-31 20:13:15.14866393 +0000 UTC m=+3188.610038587"
Oct 31 20:13:16.026435 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:16.026398    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"]
Oct 31 20:13:16.030121 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:16.030093    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a0f62c-7ksr4"]
Oct 31 20:13:16.130881 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:16.130840    5958 generic.go:356] "Generic (PLEG): container finished" podID="051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" containerID="a519fa1c428d4b03c340f72b680f70dbfccc7c3f2d94f021ba4b9858e935838c" exitCode=0
Oct 31 20:13:16.131282 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:16.130925    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h" event={"ID":"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40","Type":"ContainerDied","Data":"a519fa1c428d4b03c340f72b680f70dbfccc7c3f2d94f021ba4b9858e935838c"}
Oct 31 20:13:16.662757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:16.662725    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15e2576-4a06-4ffa-9d2c-ddc657e60da6" path="/var/lib/kubelet/pods/a15e2576-4a06-4ffa-9d2c-ddc657e60da6/volumes"
Oct 31 20:13:17.044113 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:13:17.044029    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:13:17.293055 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.293028    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:17.373293 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.373232    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca\") pod \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") "
Oct 31 20:13:17.373579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.373301    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca\") pod \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") "
Oct 31 20:13:17.373579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.373397    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw\") pod \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\" (UID: \"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40\") "
Oct 31 20:13:17.373818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.373763    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca" pod "051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" (UID: "051ded6c-6f9f-4df6-bb2d-c4b4dc803c40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:17.375694 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.375669    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw" pod "051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" (UID: "051ded6c-6f9f-4df6-bb2d-c4b4dc803c40"). InnerVolumeSpecName "kube-api-access-97nfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:17.375790 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.375765    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca" pod "051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" (UID: "051ded6c-6f9f-4df6-bb2d-c4b4dc803c40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:17.475027 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.474994    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:17.475027 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.475024    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:17.475027 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:17.475035    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40-kube-api-access-97nfw\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:18.144035 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:18.144005    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"
Oct 31 20:13:18.144209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:18.144033    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h" event={"ID":"051ded6c-6f9f-4df6-bb2d-c4b4dc803c40","Type":"ContainerDied","Data":"8de3d934474025ed04081a2c201f9f4b4bbe5c0a85c4bc72986dc781acc5c8d1"}
Oct 31 20:13:18.144209 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:18.144077    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de3d934474025ed04081a2c201f9f4b4bbe5c0a85c4bc72986dc781acc5c8d1"
Oct 31 20:13:24.091377 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.091341    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"]
Oct 31 20:13:24.092029 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.092008    5958 cpu_manager.go:414] "containerMap: removing container" podUID="051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" containerName="revoke-keys"
Oct 31 20:13:24.092145 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.092031    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" containerName="revoke-keys"
Oct 31 20:13:24.092222 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.092208    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" containerName="revoke-keys"
Oct 31 20:13:24.095782 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.095758    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.101877 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.101674    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:13:24.105398 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.105369    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:13:24.107353 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.107321    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"]
Oct 31 20:13:24.249470 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.249431    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.249693 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.249479    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.249693 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.249507    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpp8j\" (UniqueName: \"kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.351004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.350895    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.351004 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.350967    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.351265 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.351011    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpp8j\" (UniqueName: \"kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.351614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.351591    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.353361 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.353338    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.364662 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.364636    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpp8j\" (UniqueName: \"kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j\") pod \"maas-api-revoke-sub-5a29efb3-pqpgr\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.413912 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.413873    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:24.692579 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.692539    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"]
Oct 31 20:13:24.694351 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:13:24.694317    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0471f875_1d7f_4c6f_a908_9c216fdbe686.slice/crio-30aad1e5ccd2cd0df8bb2a9516813c1f0f2bf687d47ea747bb45b921416553a2 WatchSource:0}: Error finding container 30aad1e5ccd2cd0df8bb2a9516813c1f0f2bf687d47ea747bb45b921416553a2: Status 404 returned error can't find the container with id 30aad1e5ccd2cd0df8bb2a9516813c1f0f2bf687d47ea747bb45b921416553a2
Oct 31 20:13:24.784493 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:24.784458    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:13:25.188110 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:25.188072    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr" event={"ID":"0471f875-1d7f-4c6f-a908-9c216fdbe686","Type":"ContainerStarted","Data":"796507e6af24d2eac02f4461f32b092bf144c1f6df327cb3f2fd571b7e1f9bfd"}
Oct 31 20:13:25.188697 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:25.188127    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr" event={"ID":"0471f875-1d7f-4c6f-a908-9c216fdbe686","Type":"ContainerStarted","Data":"30aad1e5ccd2cd0df8bb2a9516813c1f0f2bf687d47ea747bb45b921416553a2"}
Oct 31 20:13:25.212410 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:25.212351    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr" podStartSLOduration=1.212335509 podStartE2EDuration="1.212335509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:13:25.210064371 +0000 UTC m=+3198.671438983" watchObservedRunningTime="2026-10-31 20:13:25.212335509 +0000 UTC m=+3198.673710125"
Oct 31 20:13:26.195334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:26.195300    5958 generic.go:356] "Generic (PLEG): container finished" podID="0471f875-1d7f-4c6f-a908-9c216fdbe686" containerID="796507e6af24d2eac02f4461f32b092bf144c1f6df327cb3f2fd571b7e1f9bfd" exitCode=0
Oct 31 20:13:26.195798 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:26.195390    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr" event={"ID":"0471f875-1d7f-4c6f-a908-9c216fdbe686","Type":"ContainerDied","Data":"796507e6af24d2eac02f4461f32b092bf144c1f6df327cb3f2fd571b7e1f9bfd"}
Oct 31 20:13:27.597413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.597388    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:27.694174 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.694065    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j\" (UniqueName: \"kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j\") pod \"0471f875-1d7f-4c6f-a908-9c216fdbe686\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") "
Oct 31 20:13:27.694174 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.694110    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca\") pod \"0471f875-1d7f-4c6f-a908-9c216fdbe686\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") "
Oct 31 20:13:27.694428 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.694231    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca\") pod \"0471f875-1d7f-4c6f-a908-9c216fdbe686\" (UID: \"0471f875-1d7f-4c6f-a908-9c216fdbe686\") "
Oct 31 20:13:27.694531 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.694505    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca" pod "0471f875-1d7f-4c6f-a908-9c216fdbe686" (UID: "0471f875-1d7f-4c6f-a908-9c216fdbe686"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:13:27.696377 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.696351    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca" pod "0471f875-1d7f-4c6f-a908-9c216fdbe686" (UID: "0471f875-1d7f-4c6f-a908-9c216fdbe686"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:13:27.696678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.696642    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j" pod "0471f875-1d7f-4c6f-a908-9c216fdbe686" (UID: "0471f875-1d7f-4c6f-a908-9c216fdbe686"). InnerVolumeSpecName "kube-api-access-tpp8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:13:27.796166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.796123    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:27.796166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.796163    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpp8j\" (UniqueName: \"kubernetes.io/projected/0471f875-1d7f-4c6f-a908-9c216fdbe686-kube-api-access-tpp8j\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:27.796166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:27.796173    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0471f875-1d7f-4c6f-a908-9c216fdbe686-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:13:28.208803 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:28.208768    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr" event={"ID":"0471f875-1d7f-4c6f-a908-9c216fdbe686","Type":"ContainerDied","Data":"30aad1e5ccd2cd0df8bb2a9516813c1f0f2bf687d47ea747bb45b921416553a2"}
Oct 31 20:13:28.209120 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:28.209089    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30aad1e5ccd2cd0df8bb2a9516813c1f0f2bf687d47ea747bb45b921416553a2"
Oct 31 20:13:28.209335 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:28.209173    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"
Oct 31 20:13:35.086491 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:13:35.086456    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:13:47.045398 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:13:47.045358    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:14:01.059517 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:01.059478    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"]
Oct 31 20:14:01.073067 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:01.073033    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7fbzl"]
Oct 31 20:14:02.663034 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:02.662998    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3ba027-cc50-4ccb-b674-72b52871b505" path="/var/lib/kubelet/pods/3f3ba027-cc50-4ccb-b674-72b52871b505/volumes"
Oct 31 20:14:10.239228 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:10.239190    5958 scope.go:122] "RemoveContainer" containerID="93d8cea57b366119ea846982eb67db7422c1d43d224ae43b37d2d5854af265de"
Oct 31 20:14:10.248619 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:10.248598    5958 scope.go:122] "RemoveContainer" containerID="6b8528d46783794666425413d49bdef32d452a7783d8d9abca6d6dab1e3fd0a8"
Oct 31 20:14:10.257503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:10.257479    5958 scope.go:122] "RemoveContainer" containerID="c1b5c0eb1f2a49a593e57e6aa44f6566237014e66062b1ddcf1ba9a3b2bcbb44"
Oct 31 20:14:17.047262 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:14:17.047170    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:14:47.044477 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:14:47.044440    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:14:53.028640 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:53.028595    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"]
Oct 31 20:14:53.032039 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:53.032007    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vh96n"]
Oct 31 20:14:54.663641 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:14:54.663606    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49804714-b3b3-4cee-8eec-b4bdf1abedee" path="/var/lib/kubelet/pods/49804714-b3b3-4cee-8eec-b4bdf1abedee/volumes"
Oct 31 20:15:00.161377 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.161336    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"]
Oct 31 20:15:00.162086 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.162063    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0471f875-1d7f-4c6f-a908-9c216fdbe686" containerName="revoke-keys"
Oct 31 20:15:00.162183 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.162088    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0471f875-1d7f-4c6f-a908-9c216fdbe686" containerName="revoke-keys"
Oct 31 20:15:00.162263 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.162247    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0471f875-1d7f-4c6f-a908-9c216fdbe686" containerName="revoke-keys"
Oct 31 20:15:00.165761 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.165736    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.172009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.171983    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:15:00.249274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.249237    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"]
Oct 31 20:15:00.260143 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.260113    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-jxlzk\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.260347 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.260317    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vs8\" (UniqueName: \"kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8\") pod \"maas-api-key-cleanup-29891295-jxlzk\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.361955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.361907    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-jxlzk\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.362192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.362047    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vs8\" (UniqueName: \"kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8\") pod \"maas-api-key-cleanup-29891295-jxlzk\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.362617 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.362596    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-jxlzk\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.377386 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.377354    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vs8\" (UniqueName: \"kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8\") pod \"maas-api-key-cleanup-29891295-jxlzk\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.479234 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.479140    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:00.758260 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:00.758225    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"]
Oct 31 20:15:00.759467 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:15:00.759441    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc979ee9a_7c0a_4d1f_961d_8d6ad52aa830.slice/crio-1f6e5fdf148eb9970fbc32a6ac53c4b3de545571fbb42fd0f5d74d3b4418f3d8 WatchSource:0}: Error finding container 1f6e5fdf148eb9970fbc32a6ac53c4b3de545571fbb42fd0f5d74d3b4418f3d8: Status 404 returned error can't find the container with id 1f6e5fdf148eb9970fbc32a6ac53c4b3de545571fbb42fd0f5d74d3b4418f3d8
Oct 31 20:15:01.759016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:01.758923    5958 generic.go:356] "Generic (PLEG): container finished" podID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerID="1a7b892cc287a6686bbcbeb050344389bd255c3cb1a958a93a4f9e5e0b73af83" exitCode=60
Oct 31 20:15:01.759016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:01.758975    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" event={"ID":"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830","Type":"ContainerDied","Data":"1a7b892cc287a6686bbcbeb050344389bd255c3cb1a958a93a4f9e5e0b73af83"}
Oct 31 20:15:01.759409 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:01.759025    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" event={"ID":"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830","Type":"ContainerStarted","Data":"1f6e5fdf148eb9970fbc32a6ac53c4b3de545571fbb42fd0f5d74d3b4418f3d8"}
Oct 31 20:15:01.759457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:01.759411    5958 scope.go:122] "RemoveContainer" containerID="1a7b892cc287a6686bbcbeb050344389bd255c3cb1a958a93a4f9e5e0b73af83"
Oct 31 20:15:02.767889 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:02.767796    5958 generic.go:356] "Generic (PLEG): container finished" podID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerID="3defe8e5fedd1a02574348892ffa3832ae2a3a4c4abb2a4c37b4a3368b59865a" exitCode=60
Oct 31 20:15:02.768387 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:02.767882    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" event={"ID":"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830","Type":"ContainerDied","Data":"3defe8e5fedd1a02574348892ffa3832ae2a3a4c4abb2a4c37b4a3368b59865a"}
Oct 31 20:15:02.768387 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:02.767944    5958 scope.go:122] "RemoveContainer" containerID="1a7b892cc287a6686bbcbeb050344389bd255c3cb1a958a93a4f9e5e0b73af83"
Oct 31 20:15:02.768387 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:02.768267    5958 scope.go:122] "RemoveContainer" containerID="3defe8e5fedd1a02574348892ffa3832ae2a3a4c4abb2a4c37b4a3368b59865a"
Oct 31 20:15:02.768593 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:15:02.768538    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-jxlzk_odh-ai-gateway-infra(c979ee9a-7c0a-4d1f-961d-8d6ad52aa830)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830"
Oct 31 20:15:03.775094 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:03.775063    5958 scope.go:122] "RemoveContainer" containerID="3defe8e5fedd1a02574348892ffa3832ae2a3a4c4abb2a4c37b4a3368b59865a"
Oct 31 20:15:03.775502 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:15:03.775341    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-jxlzk_odh-ai-gateway-infra(c979ee9a-7c0a-4d1f-961d-8d6ad52aa830)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830"
Oct 31 20:15:10.289722 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:10.289687    5958 scope.go:122] "RemoveContainer" containerID="55d3325f7cf3b5561158e67c50a9cde7204f1960980934e206760d751ae9ccaf"
Oct 31 20:15:10.298855 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:10.298830    5958 scope.go:122] "RemoveContainer" containerID="329c1de7127c3a12413ab4e20d223c39c882f46ed0bcc5c463fb64f7acce331a"
Oct 31 20:15:13.656816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:13.656782    5958 scope.go:122] "RemoveContainer" containerID="3defe8e5fedd1a02574348892ffa3832ae2a3a4c4abb2a4c37b4a3368b59865a"
Oct 31 20:15:14.836955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:14.836922    5958 generic.go:356] "Generic (PLEG): container finished" podID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerID="1cae08ebfab11afab68c23ce032ae037455c6bc716b9538209492db3ca1f06f6" exitCode=60
Oct 31 20:15:14.837404 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:14.836994    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" event={"ID":"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830","Type":"ContainerDied","Data":"1cae08ebfab11afab68c23ce032ae037455c6bc716b9538209492db3ca1f06f6"}
Oct 31 20:15:14.837404 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:14.837064    5958 scope.go:122] "RemoveContainer" containerID="3defe8e5fedd1a02574348892ffa3832ae2a3a4c4abb2a4c37b4a3368b59865a"
Oct 31 20:15:14.837505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:14.837433    5958 scope.go:122] "RemoveContainer" containerID="1cae08ebfab11afab68c23ce032ae037455c6bc716b9538209492db3ca1f06f6"
Oct 31 20:15:14.837767 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:15:14.837738    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-jxlzk_odh-ai-gateway-infra(c979ee9a-7c0a-4d1f-961d-8d6ad52aa830)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830"
Oct 31 20:15:15.869628 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:15.869590    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"]
Oct 31 20:15:16.012767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.012738    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:16.047357 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.047326    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"]
Oct 31 20:15:16.049226 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.049193    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5qzp9"]
Oct 31 20:15:16.128341 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.128249    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8\" (UniqueName: \"kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8\") pod \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") "
Oct 31 20:15:16.128556 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.128389    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca\") pod \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\" (UID: \"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830\") "
Oct 31 20:15:16.128750 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.128729    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca" pod "c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" (UID: "c979ee9a-7c0a-4d1f-961d-8d6ad52aa830"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:15:16.128845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.128830    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-maas-api-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:15:16.130484 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.130454    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8" pod "c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" (UID: "c979ee9a-7c0a-4d1f-961d-8d6ad52aa830"). InnerVolumeSpecName "kube-api-access-78vs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:15:16.229669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.229633    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78vs8\" (UniqueName: \"kubernetes.io/projected/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830-kube-api-access-78vs8\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:15:16.662530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.662489    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e824a49f-a4f8-4ac0-aa2f-944341f7bf12" path="/var/lib/kubelet/pods/e824a49f-a4f8-4ac0-aa2f-944341f7bf12/volumes"
Oct 31 20:15:16.851691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.851657    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"
Oct 31 20:15:16.851691 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.851683    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk" event={"ID":"c979ee9a-7c0a-4d1f-961d-8d6ad52aa830","Type":"ContainerDied","Data":"1f6e5fdf148eb9970fbc32a6ac53c4b3de545571fbb42fd0f5d74d3b4418f3d8"}
Oct 31 20:15:16.851931 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.851749    5958 scope.go:122] "RemoveContainer" containerID="1cae08ebfab11afab68c23ce032ae037455c6bc716b9538209492db3ca1f06f6"
Oct 31 20:15:16.879005 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.878977    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"]
Oct 31 20:15:16.884146 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:16.884117    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-jxlzk"]
Oct 31 20:15:17.043814 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:15:17.043733    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:15:18.662206 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:18.662172    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" path="/var/lib/kubelet/pods/c979ee9a-7c0a-4d1f-961d-8d6ad52aa830/volumes"
Oct 31 20:15:27.026429 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:27.026370    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"]
Oct 31 20:15:27.028975 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:27.028941    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bg5w6"]
Oct 31 20:15:28.662058 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:28.662014    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b895bf5-89ca-40a2-927c-aee96b6bbcc7" path="/var/lib/kubelet/pods/8b895bf5-89ca-40a2-927c-aee96b6bbcc7/volumes"
Oct 31 20:15:47.046244 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:15:47.046157    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:15:54.027404 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:54.027349    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"]
Oct 31 20:15:54.030256 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:54.030230    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-f95mg"]
Oct 31 20:15:54.662811 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:15:54.662776    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b" path="/var/lib/kubelet/pods/f0bf5a77-bdcf-468b-9ab2-c9e77d29a55b/volumes"
Oct 31 20:16:04.027943 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:04.027790    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"]
Oct 31 20:16:04.030240 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:04.030219    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5tjlh"]
Oct 31 20:16:04.663599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:04.663563    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7463aa-758b-4109-b136-68e80dea7ec4" path="/var/lib/kubelet/pods/9e7463aa-758b-4109-b136-68e80dea7ec4/volumes"
Oct 31 20:16:10.325999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:10.325963    5958 scope.go:122] "RemoveContainer" containerID="6cede83966f99e1078a892de536d9787e5ee1132e21797f08ea59673dd8fcd2d"
Oct 31 20:16:17.044428 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:16:17.044381    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:16:25.028589 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:25.028536    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"]
Oct 31 20:16:25.031334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:25.031306    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wjczb"]
Oct 31 20:16:26.662243 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:26.662200    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fe52a0-3238-48a5-bcdb-b8ac1ed79f21" path="/var/lib/kubelet/pods/23fe52a0-3238-48a5-bcdb-b8ac1ed79f21/volumes"
Oct 31 20:16:43.029275 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:43.029232    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"]
Oct 31 20:16:43.032918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:43.032887    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7fhzj"]
Oct 31 20:16:44.661892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:16:44.661850    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a841f05-b630-4be5-9a05-6f60bab75caa" path="/var/lib/kubelet/pods/7a841f05-b630-4be5-9a05-6f60bab75caa/volumes"
Oct 31 20:16:47.043947 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:16:47.043911    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:17:02.028618 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:02.028581    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"]
Oct 31 20:17:02.030690 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:02.030666    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f5tgw"]
Oct 31 20:17:02.662308 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:02.662273    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b828b17c-2771-45fc-9f70-56c71c0133d5" path="/var/lib/kubelet/pods/b828b17c-2771-45fc-9f70-56c71c0133d5/volumes"
Oct 31 20:17:10.355048 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:10.355010    5958 scope.go:122] "RemoveContainer" containerID="1ea2817ce34aeff6c381cd4c0512feb19f305aa3c4d617733ce912a99f5777e5"
Oct 31 20:17:10.364041 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:10.364017    5958 scope.go:122] "RemoveContainer" containerID="2a916ac4723ce97d14a39e77d9deb31e20cd48f6f60b963248020748b83864be"
Oct 31 20:17:10.372745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:10.372723    5958 scope.go:122] "RemoveContainer" containerID="5918de8d41381ae1896a012c8a875f049b660ca13214624102754e7baa01c6ea"
Oct 31 20:17:10.380927 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:10.380904    5958 scope.go:122] "RemoveContainer" containerID="42118ef7376c6add03b5bc8e4ad61a9e00369904b62ffa58f4853c99f2063ac2"
Oct 31 20:17:17.044438 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:17:17.044349    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:17:19.068970 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:19.068934    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"]
Oct 31 20:17:19.073407 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:19.073379    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-gdl8m"]
Oct 31 20:17:20.662460 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:20.662416    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac0f07e-da27-4fef-912e-93b448679650" path="/var/lib/kubelet/pods/2ac0f07e-da27-4fef-912e-93b448679650/volumes"
Oct 31 20:17:47.044491 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:17:47.044454    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:17:49.029043 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:49.029002    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"]
Oct 31 20:17:49.031565 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:49.031530    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-x6wpz"]
Oct 31 20:17:50.662785 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:17:50.662752    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e76b29-cec5-465e-8da7-3f8e6323cd69" path="/var/lib/kubelet/pods/28e76b29-cec5-465e-8da7-3f8e6323cd69/volumes"
Oct 31 20:18:10.415397 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:10.415365    5958 scope.go:122] "RemoveContainer" containerID="67760fe95d6c85a46ed562b36e58ac6f26e9551454af1f191c5ad6ccfa4adfb5"
Oct 31 20:18:10.424900 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:10.424875    5958 scope.go:122] "RemoveContainer" containerID="46f303e1740c36b88d614d0dbe3f63908277023f26bb8b9d03561d7f1cb8b996"
Oct 31 20:18:10.433198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:10.433175    5958 scope.go:122] "RemoveContainer" containerID="bc3ccf7bd9bbe1f510bb6eca672988ccce8fb6eca7729e4842a6fff02d2df707"
Oct 31 20:18:17.044953 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:18:17.044912    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:18:18.026602 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:18.026560    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"]
Oct 31 20:18:18.031673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:18.031638    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6l9h"]
Oct 31 20:18:18.661989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:18.661953    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051ded6c-6f9f-4df6-bb2d-c4b4dc803c40" path="/var/lib/kubelet/pods/051ded6c-6f9f-4df6-bb2d-c4b4dc803c40/volumes"
Oct 31 20:18:28.027172 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:28.027126    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"]
Oct 31 20:18:28.030223 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:28.030193    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pqpgr"]
Oct 31 20:18:28.662471 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:28.662437    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0471f875-1d7f-4c6f-a908-9c216fdbe686" path="/var/lib/kubelet/pods/0471f875-1d7f-4c6f-a908-9c216fdbe686/volumes"
Oct 31 20:18:39.956470 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.956431    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"]
Oct 31 20:18:39.957132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957110    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957134    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957167    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957175    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957187    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957232 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957195    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957311    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.957514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.957325    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:18:39.960621 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.960601    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:39.967286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.967260    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:18:39.967457 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.967285    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:18:39.995440 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:39.995400    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"]
Oct 31 20:18:40.009332 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.009288    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xfh\" (UniqueName: \"kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.009562 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.009344    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.009562 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.009506    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.110878 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.110838    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.111112 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.110920    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xfh\" (UniqueName: \"kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.111112 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.110960    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.111770 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.111744    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.113392 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.113366    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.124481 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.124450    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xfh\" (UniqueName: \"kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh\") pod \"maas-api-revoke-sub-3d4156f1-rtqjg\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.273809 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.273729    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:40.548288 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.548262    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"]
Oct 31 20:18:40.550072 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:18:40.550041    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7f11ec_d65a_46cb_937d_c0ccbe305459.slice/crio-4462db56b39ee29150004bb5b63e7e7cc36e8e8b143de18d4a1914c9a5ebe4c7 WatchSource:0}: Error finding container 4462db56b39ee29150004bb5b63e7e7cc36e8e8b143de18d4a1914c9a5ebe4c7: Status 404 returned error can't find the container with id 4462db56b39ee29150004bb5b63e7e7cc36e8e8b143de18d4a1914c9a5ebe4c7
Oct 31 20:18:40.971597 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.971533    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg" event={"ID":"3e7f11ec-d65a-46cb-937d-c0ccbe305459","Type":"ContainerStarted","Data":"a7c7cb814beb654eb0ef116e05bacd238af52384da7c216cf716fb49523065a5"}
Oct 31 20:18:40.972000 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:40.971620    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg" event={"ID":"3e7f11ec-d65a-46cb-937d-c0ccbe305459","Type":"ContainerStarted","Data":"4462db56b39ee29150004bb5b63e7e7cc36e8e8b143de18d4a1914c9a5ebe4c7"}
Oct 31 20:18:41.978449 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:41.978408    5958 generic.go:356] "Generic (PLEG): container finished" podID="3e7f11ec-d65a-46cb-937d-c0ccbe305459" containerID="a7c7cb814beb654eb0ef116e05bacd238af52384da7c216cf716fb49523065a5" exitCode=0
Oct 31 20:18:41.978915 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:41.978489    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg" event={"ID":"3e7f11ec-d65a-46cb-937d-c0ccbe305459","Type":"ContainerDied","Data":"a7c7cb814beb654eb0ef116e05bacd238af52384da7c216cf716fb49523065a5"}
Oct 31 20:18:43.120300 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.120274    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:43.243920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.243821    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh\" (UniqueName: \"kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh\") pod \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") "
Oct 31 20:18:43.243920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.243863    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca\") pod \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") "
Oct 31 20:18:43.244196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.243935    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca\") pod \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\" (UID: \"3e7f11ec-d65a-46cb-937d-c0ccbe305459\") "
Oct 31 20:18:43.244331 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.244309    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca" pod "3e7f11ec-d65a-46cb-937d-c0ccbe305459" (UID: "3e7f11ec-d65a-46cb-937d-c0ccbe305459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:18:43.246093 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.246066    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh" pod "3e7f11ec-d65a-46cb-937d-c0ccbe305459" (UID: "3e7f11ec-d65a-46cb-937d-c0ccbe305459"). InnerVolumeSpecName "kube-api-access-h2xfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:18:43.246213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.246097    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca" pod "3e7f11ec-d65a-46cb-937d-c0ccbe305459" (UID: "3e7f11ec-d65a-46cb-937d-c0ccbe305459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:18:43.345289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.345252    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:18:43.345289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.345283    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7f11ec-d65a-46cb-937d-c0ccbe305459-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:18:43.345289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.345294    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xfh\" (UniqueName: \"kubernetes.io/projected/3e7f11ec-d65a-46cb-937d-c0ccbe305459-kube-api-access-h2xfh\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:18:43.989324 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.989294    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"
Oct 31 20:18:43.989514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.989335    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg" event={"ID":"3e7f11ec-d65a-46cb-937d-c0ccbe305459","Type":"ContainerDied","Data":"4462db56b39ee29150004bb5b63e7e7cc36e8e8b143de18d4a1914c9a5ebe4c7"}
Oct 31 20:18:43.989514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:43.989384    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4462db56b39ee29150004bb5b63e7e7cc36e8e8b143de18d4a1914c9a5ebe4c7"
Oct 31 20:18:47.047511 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:18:47.047438    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:18:50.785207 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:50.785172    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:18:54.779381 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:54.779345    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:18:55.079505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:55.079405    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"]
Oct 31 20:18:55.079741 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:18:55.079670    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy" containerID="cri-o://8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88" gracePeriod=30
Oct 31 20:19:00.342248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.342219    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:19:00.384414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384376    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384445    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384489    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384530    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384589    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384629    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384747    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384794    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert\") pod \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\" (UID: \"2070b9bd-6ab1-4d9f-be48-3200efc1b85e\") "
Oct 31 20:19:00.384963 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384838    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.385184 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.384991    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.385245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.385222    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-credential-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.385304 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.385246    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-data\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.385362 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.385298    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:19:00.385446 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.385411    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.385527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.385483    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.387341 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.387314    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.387511 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.387424    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:19:00.387620 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.387514    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.387800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.387775    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn" pod "2070b9bd-6ab1-4d9f-be48-3200efc1b85e" (UID: "2070b9bd-6ab1-4d9f-be48-3200efc1b85e"). InnerVolumeSpecName "kube-api-access-kj8wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.485972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.485908    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-podinfo\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.485972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.485950    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.485972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.485962    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-token\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.485972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.485974    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istiod-ca-cert\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.486287 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.485986    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-istio-envoy\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.486287 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.485999    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj8wn\" (UniqueName: \"kubernetes.io/projected/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-kube-api-access-kj8wn\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.486287 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:00.486015    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2070b9bd-6ab1-4d9f-be48-3200efc1b85e-workload-socket\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:01.117254 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.117217    5958 generic.go:356] "Generic (PLEG): container finished" podID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerID="8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88" exitCode=0
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.117294    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.117314    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" event={"ID":"2070b9bd-6ab1-4d9f-be48-3200efc1b85e","Type":"ContainerDied","Data":"8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88"}
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.117373    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf" event={"ID":"2070b9bd-6ab1-4d9f-be48-3200efc1b85e","Type":"ContainerDied","Data":"a8ee374b6def543bf8d99c22e97662218ba552ffa3e7b11f806d2029d3cc4b8e"}
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.117399    5958 scope.go:122] "RemoveContainer" containerID="8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88"
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.129678    5958 scope.go:122] "RemoveContainer" containerID="8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88"
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:19:01.130074    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88\": container with ID starting with 8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88 not found: ID does not exist" containerID="8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88"
Oct 31 20:19:01.136468 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.130110    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88"} err="failed to get container status \"8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88\": rpc error: code = NotFound desc = could not find container \"8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88\": container with ID starting with 8a8f7af636fead6dca095b118f8bcb9c0ca876acf0e99a1039d84d1c61ce1b88 not found: ID does not exist"
Oct 31 20:19:01.148583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.148530    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"]
Oct 31 20:19:01.152701 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:01.152674    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6ncmf"]
Oct 31 20:19:02.662617 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:02.662586    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" path="/var/lib/kubelet/pods/2070b9bd-6ab1-4d9f-be48-3200efc1b85e/volumes"
Oct 31 20:19:05.226512 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:05.226475    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:19:10.463255 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:10.463220    5958 scope.go:122] "RemoveContainer" containerID="6a4ae94cd788d0401ca27647a7b06edd39fd89b69cca6b2474de6f2414dc2c47"
Oct 31 20:19:10.471972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:10.471951    5958 scope.go:122] "RemoveContainer" containerID="56234798b70ca104e412629332d572da6a31bce4be7e126ca67b0989574f6e7b"
Oct 31 20:19:10.480825 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:10.480801    5958 scope.go:122] "RemoveContainer" containerID="239e3a3c8de5d7e0e9b4e989cff33e1f17c9db2b5fd4d344510fe73b4fe9ac3b"
Oct 31 20:19:17.053822 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:19:17.053783    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:19:20.673205 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:20.673168    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:19:37.512271 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:37.512224    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:19:40.248558 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.248510    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"]
Oct 31 20:19:40.249800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.249770    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3e7f11ec-d65a-46cb-937d-c0ccbe305459" containerName="revoke-keys"
Oct 31 20:19:40.250001 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.249985    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7f11ec-d65a-46cb-937d-c0ccbe305459" containerName="revoke-keys"
Oct 31 20:19:40.250165 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.250153    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy"
Oct 31 20:19:40.250248 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.250237    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy"
Oct 31 20:19:40.250652 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.250634    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7f11ec-d65a-46cb-937d-c0ccbe305459" containerName="revoke-keys"
Oct 31 20:19:40.250792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.250781    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2070b9bd-6ab1-4d9f-be48-3200efc1b85e" containerName="istio-proxy"
Oct 31 20:19:40.250885 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.250876    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c979ee9a-7c0a-4d1f-961d-8d6ad52aa830" containerName="cleanup"
Oct 31 20:19:40.263894 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.263859    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.275218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.275181    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"]
Oct 31 20:19:40.410527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410484    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.410777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410581    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbtq\" (UniqueName: \"kubernetes.io/projected/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-kube-api-access-wzbtq\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.410777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410629    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.410777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410699    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.410777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410770    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.410946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410843    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.410946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410916    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.411036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410953    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.411036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.410971    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512107    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512160    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512193    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbtq\" (UniqueName: \"kubernetes.io/projected/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-kube-api-access-wzbtq\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512213 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512217    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512239    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512307    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512412    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512451    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512936 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512825    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.512936 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512854    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.513062 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512925    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.513062 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.512964    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.513184 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.513151    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.514720 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.514698    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.514860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.514843    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.526190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.526165    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.526321 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.526253    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbtq\" (UniqueName: \"kubernetes.io/projected/f4d48a85-41c3-4baf-9bcd-1bc0af46856d-kube-api-access-wzbtq\") pod \"maas-default-gateway-openshift-default-75d7c785f-pdvwj\" (UID: \"f4d48a85-41c3-4baf-9bcd-1bc0af46856d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.584422 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.584380    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:40.869690 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.869659    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"]
Oct 31 20:19:40.871472 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:19:40.871446    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d48a85_41c3_4baf_9bcd_1bc0af46856d.slice/crio-f6422a1a10c9969230c77de7726b862121db8b7f61ee2993d85a084e03c8fd60 WatchSource:0}: Error finding container f6422a1a10c9969230c77de7726b862121db8b7f61ee2993d85a084e03c8fd60: Status 404 returned error can't find the container with id f6422a1a10c9969230c77de7726b862121db8b7f61ee2993d85a084e03c8fd60
Oct 31 20:19:40.873709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.873679    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:19:40.873820 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.873790    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:19:40.873886 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:40.873836    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 20:19:41.337897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:41.337803    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" event={"ID":"f4d48a85-41c3-4baf-9bcd-1bc0af46856d","Type":"ContainerStarted","Data":"b4ab62012e38db4e957f1e5f046ac7bab2e0d5b0d6c2b0bc9e4acbc911076623"}
Oct 31 20:19:41.337897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:41.337867    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" event={"ID":"f4d48a85-41c3-4baf-9bcd-1bc0af46856d","Type":"ContainerStarted","Data":"f6422a1a10c9969230c77de7726b862121db8b7f61ee2993d85a084e03c8fd60"}
Oct 31 20:19:41.367192 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:41.367127    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podStartSLOduration=1.367107144 podStartE2EDuration="1.367107144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:41.364892559 +0000 UTC m=+3574.826267186" watchObservedRunningTime="2026-10-31 20:19:41.367107144 +0000 UTC m=+3574.828481762"
Oct 31 20:19:41.584729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:41.584692    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:41.586273 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:41.586243    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:41.586421 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:41.586298    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:42.585726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:42.585688    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:42.586141 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:42.585742    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:43.585559 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:43.585524    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:43.585754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:43.585621    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:44.585811 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:44.585772    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:44.586200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:44.585845    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:45.585252 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:45.585209    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:45.585456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:45.585284    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:46.585775 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:46.585740    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:46.586200 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:46.585797    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:47.045216 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:19:47.045173    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:19:47.585312 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:47.585271    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-pdvwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body=
Oct 31 20:19:47.585537 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:47.585354    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj" podUID="f4d48a85-41c3-4baf-9bcd-1bc0af46856d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused"
Oct 31 20:19:49.259671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.259636    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:49.260252 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.260128    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:49.261319 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.261295    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-pdvwj"
Oct 31 20:19:49.326684 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.326621    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"]
Oct 31 20:19:49.332296 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.332268    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.340010 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.339972    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"]
Oct 31 20:19:49.340196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.340102    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:19:49.340196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.340160    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:19:49.421027 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.420987    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvh9t\" (UniqueName: \"kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.421358 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.421325    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.421673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.421537    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.522817 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.522726    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvh9t\" (UniqueName: \"kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.523800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.523774    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.524919 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.524898    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.526297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.526132    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.533030 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.533000    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.542178 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.542107    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvh9t\" (UniqueName: \"kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t\") pod \"maas-api-revoke-sub-ba3384e0-mlwj6\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.656824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.656783    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:49.930839 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:49.930812    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"]
Oct 31 20:19:49.933055 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:19:49.933023    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453b70ce_ca12_4975_af54_0ef5f6026c9b.slice/crio-41f9d4273f122fdc3eb7a0947d837bf74d40f657dbd16f56804bcddb81fead94 WatchSource:0}: Error finding container 41f9d4273f122fdc3eb7a0947d837bf74d40f657dbd16f56804bcddb81fead94: Status 404 returned error can't find the container with id 41f9d4273f122fdc3eb7a0947d837bf74d40f657dbd16f56804bcddb81fead94
Oct 31 20:19:50.394016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:50.393977    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6" event={"ID":"453b70ce-ca12-4975-af54-0ef5f6026c9b","Type":"ContainerStarted","Data":"1b790c7cca8dff5dc303a5b7b558e64ac016e888fc226c6a925ba3f3b4df545a"}
Oct 31 20:19:50.394494 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:50.394074    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6" event={"ID":"453b70ce-ca12-4975-af54-0ef5f6026c9b","Type":"ContainerStarted","Data":"41f9d4273f122fdc3eb7a0947d837bf74d40f657dbd16f56804bcddb81fead94"}
Oct 31 20:19:50.415009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:50.414959    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6" podStartSLOduration=1.414945083 podStartE2EDuration="1.414945083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:50.414619236 +0000 UTC m=+3583.875993850" watchObservedRunningTime="2026-10-31 20:19:50.414945083 +0000 UTC m=+3583.876319698"
Oct 31 20:19:51.400996 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:51.400961    5958 generic.go:356] "Generic (PLEG): container finished" podID="453b70ce-ca12-4975-af54-0ef5f6026c9b" containerID="1b790c7cca8dff5dc303a5b7b558e64ac016e888fc226c6a925ba3f3b4df545a" exitCode=0
Oct 31 20:19:51.401423 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:51.401024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6" event={"ID":"453b70ce-ca12-4975-af54-0ef5f6026c9b","Type":"ContainerDied","Data":"1b790c7cca8dff5dc303a5b7b558e64ac016e888fc226c6a925ba3f3b4df545a"}
Oct 31 20:19:52.543340 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.543315    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:52.660462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.660371    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca\") pod \"453b70ce-ca12-4975-af54-0ef5f6026c9b\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") "
Oct 31 20:19:52.660462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.660459    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t\" (UniqueName: \"kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t\") pod \"453b70ce-ca12-4975-af54-0ef5f6026c9b\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") "
Oct 31 20:19:52.660732 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.660500    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca\") pod \"453b70ce-ca12-4975-af54-0ef5f6026c9b\" (UID: \"453b70ce-ca12-4975-af54-0ef5f6026c9b\") "
Oct 31 20:19:52.660844 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.660817    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca" pod "453b70ce-ca12-4975-af54-0ef5f6026c9b" (UID: "453b70ce-ca12-4975-af54-0ef5f6026c9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:19:52.660964 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.660946    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:52.662702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.662677    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca" pod "453b70ce-ca12-4975-af54-0ef5f6026c9b" (UID: "453b70ce-ca12-4975-af54-0ef5f6026c9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:19:52.662702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.662684    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t" pod "453b70ce-ca12-4975-af54-0ef5f6026c9b" (UID: "453b70ce-ca12-4975-af54-0ef5f6026c9b"). InnerVolumeSpecName "kube-api-access-fvh9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:52.762429 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.762390    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvh9t\" (UniqueName: \"kubernetes.io/projected/453b70ce-ca12-4975-af54-0ef5f6026c9b-kube-api-access-fvh9t\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:52.762429 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:52.762427    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453b70ce-ca12-4975-af54-0ef5f6026c9b-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:53.413250 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:53.413219    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"
Oct 31 20:19:53.413443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:53.413247    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6" event={"ID":"453b70ce-ca12-4975-af54-0ef5f6026c9b","Type":"ContainerDied","Data":"41f9d4273f122fdc3eb7a0947d837bf74d40f657dbd16f56804bcddb81fead94"}
Oct 31 20:19:53.413443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:19:53.413297    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f9d4273f122fdc3eb7a0947d837bf74d40f657dbd16f56804bcddb81fead94"
Oct 31 20:20:00.455830 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.455789    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"]
Oct 31 20:20:00.456560 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.456509    5958 cpu_manager.go:414] "containerMap: removing container" podUID="453b70ce-ca12-4975-af54-0ef5f6026c9b" containerName="revoke-keys"
Oct 31 20:20:00.456677 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.456560    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453b70ce-ca12-4975-af54-0ef5f6026c9b" containerName="revoke-keys"
Oct 31 20:20:00.456725 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.456692    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="453b70ce-ca12-4975-af54-0ef5f6026c9b" containerName="revoke-keys"
Oct 31 20:20:00.459891 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.459870    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.467578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.467503    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:20:00.467749 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.467538    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:20:00.470236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.470204    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"]
Oct 31 20:20:00.553461 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.553418    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.553689 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.553572    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k426w\" (UniqueName: \"kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.553689 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.553676    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.655362 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.655315    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.655613 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.655432    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k426w\" (UniqueName: \"kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.655613 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.655511    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.656230 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.656197    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.658121 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.658092    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.669768 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.669732    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k426w\" (UniqueName: \"kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w\") pod \"maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:00.789603 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:00.789495    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:01.067132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:01.067102    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"]
Oct 31 20:20:01.069114 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:20:01.069082    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5dc2d5_ca5a_4aaa_a1bb_edcb85f8ab6a.slice/crio-527bf07470247d693806552989bab28d384626f27f709c9b7b63ada614971fb7 WatchSource:0}: Error finding container 527bf07470247d693806552989bab28d384626f27f709c9b7b63ada614971fb7: Status 404 returned error can't find the container with id 527bf07470247d693806552989bab28d384626f27f709c9b7b63ada614971fb7
Oct 31 20:20:01.205590 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:01.205538    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:20:01.458719 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:01.458680    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g" event={"ID":"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a","Type":"ContainerStarted","Data":"b4cf6ad1658687fc17745436c039f23fa9814165dc06e63b9c11a250d7ccffa0"}
Oct 31 20:20:01.459156 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:01.458740    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g" event={"ID":"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a","Type":"ContainerStarted","Data":"527bf07470247d693806552989bab28d384626f27f709c9b7b63ada614971fb7"}
Oct 31 20:20:01.492576 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:01.491659    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g" podStartSLOduration=1.491636623 podStartE2EDuration="1.491636623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:20:01.48170174 +0000 UTC m=+3594.943076356" watchObservedRunningTime="2026-10-31 20:20:01.491636623 +0000 UTC m=+3594.953011238"
Oct 31 20:20:02.465635 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:02.465600    5958 generic.go:356] "Generic (PLEG): container finished" podID="3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" containerID="b4cf6ad1658687fc17745436c039f23fa9814165dc06e63b9c11a250d7ccffa0" exitCode=0
Oct 31 20:20:02.466111 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:02.465704    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g" event={"ID":"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a","Type":"ContainerDied","Data":"b4cf6ad1658687fc17745436c039f23fa9814165dc06e63b9c11a250d7ccffa0"}
Oct 31 20:20:03.636491 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.636462    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:03.688222 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.688182    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca\") pod \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") "
Oct 31 20:20:03.688426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.688314    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w\" (UniqueName: \"kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w\") pod \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") "
Oct 31 20:20:03.688426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.688372    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca\") pod \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\" (UID: \"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a\") "
Oct 31 20:20:03.688637 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.688615    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca" pod "3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" (UID: "3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:20:03.688757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.688736    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:03.690462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.690439    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w" pod "3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" (UID: "3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a"). InnerVolumeSpecName "kube-api-access-k426w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:03.690559 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.690514    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca" pod "3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" (UID: "3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:20:03.789611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.789481    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k426w\" (UniqueName: \"kubernetes.io/projected/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-kube-api-access-k426w\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:03.789611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:03.789520    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:04.477616 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:04.477578    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"
Oct 31 20:20:04.477836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:04.477610    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g" event={"ID":"3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a","Type":"ContainerDied","Data":"527bf07470247d693806552989bab28d384626f27f709c9b7b63ada614971fb7"}
Oct 31 20:20:04.477836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:04.477682    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527bf07470247d693806552989bab28d384626f27f709c9b7b63ada614971fb7"
Oct 31 20:20:10.514216 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:10.514178    5958 scope.go:122] "RemoveContainer" containerID="a519fa1c428d4b03c340f72b680f70dbfccc7c3f2d94f021ba4b9858e935838c"
Oct 31 20:20:10.523489 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:10.523465    5958 scope.go:122] "RemoveContainer" containerID="796507e6af24d2eac02f4461f32b092bf144c1f6df327cb3f2fd571b7e1f9bfd"
Oct 31 20:20:11.983910 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:11.983873    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"]
Oct 31 20:20:11.987417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:11.987389    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-844db7-2ks4g"]
Oct 31 20:20:12.662701 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:12.662666    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" path="/var/lib/kubelet/pods/3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a/volumes"
Oct 31 20:20:13.327703 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:13.327664    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:20:17.045622 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:20:17.045518    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:20:20.667257 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.667228    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"]
Oct 31 20:20:20.668838 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.668814    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rtqjg"]
Oct 31 20:20:20.708640 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.708604    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"]
Oct 31 20:20:20.709185 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.709164    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" containerName="revoke-keys"
Oct 31 20:20:20.709290 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.709189    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" containerName="revoke-keys"
Oct 31 20:20:20.709350 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.709311    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5dc2d5-ca5a-4aaa-a1bb-edcb85f8ab6a" containerName="revoke-keys"
Oct 31 20:20:20.714169 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.714142    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.721076 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.721044    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"]
Oct 31 20:20:20.750417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.750372    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:20:20.750696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.750446    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:20:20.878800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.878760    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.878800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.878805    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.879037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.878899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gk9v\" (UniqueName: \"kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.979896 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.979806    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gk9v\" (UniqueName: \"kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.980068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.979931    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.980068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.979953    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.980643 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.980614    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.982422 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.982401    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:20.993453 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:20.993429    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gk9v\" (UniqueName: \"kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v\") pod \"maas-api-revoke-sub-3d4156f1-9zt2n\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:21.030294 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:21.030257    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:21.306922 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:21.306890    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"]
Oct 31 20:20:21.308922 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:20:21.308895    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbdbfee_7809_4798_bf05_d95dd3ee166a.slice/crio-5794fd39b460e59ed96d663308635c082b2187d80d39472d67a8038f4cf8d353 WatchSource:0}: Error finding container 5794fd39b460e59ed96d663308635c082b2187d80d39472d67a8038f4cf8d353: Status 404 returned error can't find the container with id 5794fd39b460e59ed96d663308635c082b2187d80d39472d67a8038f4cf8d353
Oct 31 20:20:21.586051 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:21.585962    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n" event={"ID":"3dbdbfee-7809-4798-bf05-d95dd3ee166a","Type":"ContainerStarted","Data":"5794fd39b460e59ed96d663308635c082b2187d80d39472d67a8038f4cf8d353"}
Oct 31 20:20:22.138396 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:20:22.138360    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbdbfee_7809_4798_bf05_d95dd3ee166a.slice/crio-conmon-44a3922c8325d170afac627d4ddca4b9858bcf81e0007120d5ba540dc1884d51.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:20:22.592333 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:22.592292    5958 generic.go:356] "Generic (PLEG): container finished" podID="3dbdbfee-7809-4798-bf05-d95dd3ee166a" containerID="44a3922c8325d170afac627d4ddca4b9858bcf81e0007120d5ba540dc1884d51" exitCode=0
Oct 31 20:20:22.592594 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:22.592374    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n" event={"ID":"3dbdbfee-7809-4798-bf05-d95dd3ee166a","Type":"ContainerDied","Data":"44a3922c8325d170afac627d4ddca4b9858bcf81e0007120d5ba540dc1884d51"}
Oct 31 20:20:22.663237 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:22.663199    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7f11ec-d65a-46cb-937d-c0ccbe305459" path="/var/lib/kubelet/pods/3e7f11ec-d65a-46cb-937d-c0ccbe305459/volumes"
Oct 31 20:20:23.740995 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.740967    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:23.909535 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.909497    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca\") pod \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") "
Oct 31 20:20:23.909758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.909606    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca\") pod \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") "
Oct 31 20:20:23.909758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.909742    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v\" (UniqueName: \"kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v\") pod \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\" (UID: \"3dbdbfee-7809-4798-bf05-d95dd3ee166a\") "
Oct 31 20:20:23.910009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.909986    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca" pod "3dbdbfee-7809-4798-bf05-d95dd3ee166a" (UID: "3dbdbfee-7809-4798-bf05-d95dd3ee166a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:20:23.911718 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.911696    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v" pod "3dbdbfee-7809-4798-bf05-d95dd3ee166a" (UID: "3dbdbfee-7809-4798-bf05-d95dd3ee166a"). InnerVolumeSpecName "kube-api-access-8gk9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:23.911718 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:23.911707    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca" pod "3dbdbfee-7809-4798-bf05-d95dd3ee166a" (UID: "3dbdbfee-7809-4798-bf05-d95dd3ee166a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:20:24.011131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:24.011084    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.011131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:24.011124    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbdbfee-7809-4798-bf05-d95dd3ee166a-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.011131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:24.011140    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gk9v\" (UniqueName: \"kubernetes.io/projected/3dbdbfee-7809-4798-bf05-d95dd3ee166a-kube-api-access-8gk9v\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.605107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:24.605069    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"
Oct 31 20:20:24.605107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:24.605099    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n" event={"ID":"3dbdbfee-7809-4798-bf05-d95dd3ee166a","Type":"ContainerDied","Data":"5794fd39b460e59ed96d663308635c082b2187d80d39472d67a8038f4cf8d353"}
Oct 31 20:20:24.605335 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:24.605175    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5794fd39b460e59ed96d663308635c082b2187d80d39472d67a8038f4cf8d353"
Oct 31 20:20:32.147320 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:32.147285    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:20:37.211334 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:20:37.211291    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:20:47.048686 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:20:47.048628    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:21:17.044395 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:21:17.044344    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:21:47.046023 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:21:47.045937    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:22:02.392383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:02.392343    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:22:06.509794 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.509756    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"]
Oct 31 20:22:06.510270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.510256    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3dbdbfee-7809-4798-bf05-d95dd3ee166a" containerName="revoke-keys"
Oct 31 20:22:06.510335 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.510271    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbdbfee-7809-4798-bf05-d95dd3ee166a" containerName="revoke-keys"
Oct 31 20:22:06.510375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.510356    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbdbfee-7809-4798-bf05-d95dd3ee166a" containerName="revoke-keys"
Oct 31 20:22:06.513844 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.513821    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.518435 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.518409    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:22:06.520679 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.520651    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:22:06.524250 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.524226    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"]
Oct 31 20:22:06.641902 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.641867    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.641902 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.641908    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.642129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.641994    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzg4\" (UniqueName: \"kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.742676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.742633    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.742907 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.742689    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.742907 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.742806    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzg4\" (UniqueName: \"kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.743403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.743381    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.746524 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.746504    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:22:06.756002 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.755972    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.756129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.756111    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzg4\" (UniqueName: \"kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4\") pod \"maas-api-revoke-sub-d576e68b-s6slj\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:06.837113 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.837036    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:22:06.843823 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:06.843776    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:07.122088 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:07.122055    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"]
Oct 31 20:22:07.123929 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:22:07.123901    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98c0ef0_03d4_4b05_90ad_d794057f06bb.slice/crio-f15c989a7f8e7fb1d90d84cdbbe40c95b470651ab713ceba9923d6523392f9a2 WatchSource:0}: Error finding container f15c989a7f8e7fb1d90d84cdbbe40c95b470651ab713ceba9923d6523392f9a2: Status 404 returned error can't find the container with id f15c989a7f8e7fb1d90d84cdbbe40c95b470651ab713ceba9923d6523392f9a2
Oct 31 20:22:07.178139 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:07.178105    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj" event={"ID":"b98c0ef0-03d4-4b05-90ad-d794057f06bb","Type":"ContainerStarted","Data":"f15c989a7f8e7fb1d90d84cdbbe40c95b470651ab713ceba9923d6523392f9a2"}
Oct 31 20:22:08.184989 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:08.184905    5958 generic.go:356] "Generic (PLEG): container finished" podID="b98c0ef0-03d4-4b05-90ad-d794057f06bb" containerID="9fa8ae70e56788fe93c14cb4f0d71100f6351693fce00edf4bed3fa1a2e80dad" exitCode=0
Oct 31 20:22:08.185378 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:08.184988    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj" event={"ID":"b98c0ef0-03d4-4b05-90ad-d794057f06bb","Type":"ContainerDied","Data":"9fa8ae70e56788fe93c14cb4f0d71100f6351693fce00edf4bed3fa1a2e80dad"}
Oct 31 20:22:09.333608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.333583    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:09.474523 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.474435    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca\") pod \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") "
Oct 31 20:22:09.474745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.474536    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca\") pod \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") "
Oct 31 20:22:09.474745 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.474632    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4\" (UniqueName: \"kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4\") pod \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\" (UID: \"b98c0ef0-03d4-4b05-90ad-d794057f06bb\") "
Oct 31 20:22:09.474965 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.474939    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca" pod "b98c0ef0-03d4-4b05-90ad-d794057f06bb" (UID: "b98c0ef0-03d4-4b05-90ad-d794057f06bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:22:09.476534 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.476513    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca" pod "b98c0ef0-03d4-4b05-90ad-d794057f06bb" (UID: "b98c0ef0-03d4-4b05-90ad-d794057f06bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:22:09.476680 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.476663    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4" pod "b98c0ef0-03d4-4b05-90ad-d794057f06bb" (UID: "b98c0ef0-03d4-4b05-90ad-d794057f06bb"). InnerVolumeSpecName "kube-api-access-pgzg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:22:09.575714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.575678    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:22:09.575714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.575707    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b98c0ef0-03d4-4b05-90ad-d794057f06bb-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:22:09.575714 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:09.575718    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgzg4\" (UniqueName: \"kubernetes.io/projected/b98c0ef0-03d4-4b05-90ad-d794057f06bb-kube-api-access-pgzg4\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:22:10.196537 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:10.196505    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"
Oct 31 20:22:10.196537 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:10.196522    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj" event={"ID":"b98c0ef0-03d4-4b05-90ad-d794057f06bb","Type":"ContainerDied","Data":"f15c989a7f8e7fb1d90d84cdbbe40c95b470651ab713ceba9923d6523392f9a2"}
Oct 31 20:22:10.196790 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:10.196582    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f15c989a7f8e7fb1d90d84cdbbe40c95b470651ab713ceba9923d6523392f9a2"
Oct 31 20:22:17.044366 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:22:17.044319    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:22:17.411338 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:17.411301    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:22:20.999160 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:20.999123    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:22:25.516030 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.515991    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"]
Oct 31 20:22:25.516597 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.516576    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b98c0ef0-03d4-4b05-90ad-d794057f06bb" containerName="revoke-keys"
Oct 31 20:22:25.516688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.516598    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98c0ef0-03d4-4b05-90ad-d794057f06bb" containerName="revoke-keys"
Oct 31 20:22:25.516741 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.516731    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98c0ef0-03d4-4b05-90ad-d794057f06bb" containerName="revoke-keys"
Oct 31 20:22:25.520331 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.520306    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.533659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.533625    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"]
Oct 31 20:22:25.533929 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.533911    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:22:25.533984 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.533975    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:22:25.567991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.567960    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htk8v\" (UniqueName: \"kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.567991 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.567994    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.568211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.568140    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.669737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.669696    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htk8v\" (UniqueName: \"kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.669737 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.669738    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.669972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.669784    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.670518 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.670493    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.672224 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.672201    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.688726 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.688697    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htk8v\" (UniqueName: \"kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v\") pod \"maas-api-revoke-sub-d2bc93bc-ldvjw\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:25.835450 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:25.835354    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:26.114055 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:26.114029    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"]
Oct 31 20:22:26.115430 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:22:26.115402    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c880c4a_ea6b_4d0b_a89b_6f42ce22b524.slice/crio-1019c5ed48e66cc8a55d7c536e7c7f152dfb6c7f6acca4c45adbb9b85f0925c3 WatchSource:0}: Error finding container 1019c5ed48e66cc8a55d7c536e7c7f152dfb6c7f6acca4c45adbb9b85f0925c3: Status 404 returned error can't find the container with id 1019c5ed48e66cc8a55d7c536e7c7f152dfb6c7f6acca4c45adbb9b85f0925c3
Oct 31 20:22:26.298617 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:26.298575    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw" event={"ID":"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524","Type":"ContainerStarted","Data":"1019c5ed48e66cc8a55d7c536e7c7f152dfb6c7f6acca4c45adbb9b85f0925c3"}
Oct 31 20:22:27.306015 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:27.305925    5958 generic.go:356] "Generic (PLEG): container finished" podID="7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" containerID="1e244da13bea1c63de3c42e5bf98de13f0c504ddc9bd993dc8726b37ff5f466b" exitCode=0
Oct 31 20:22:27.306404 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:27.306010    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw" event={"ID":"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524","Type":"ContainerDied","Data":"1e244da13bea1c63de3c42e5bf98de13f0c504ddc9bd993dc8726b37ff5f466b"}
Oct 31 20:22:28.447131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.447099    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:28.498653 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.498615    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca\") pod \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") "
Oct 31 20:22:28.498839 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.498729    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca\") pod \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") "
Oct 31 20:22:28.498839 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.498759    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v\" (UniqueName: \"kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v\") pod \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\" (UID: \"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524\") "
Oct 31 20:22:28.499011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.498987    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca" pod "7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" (UID: "7c880c4a-ea6b-4d0b-a89b-6f42ce22b524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:22:28.500860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.500830    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca" pod "7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" (UID: "7c880c4a-ea6b-4d0b-a89b-6f42ce22b524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:22:28.500967 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.500881    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v" pod "7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" (UID: "7c880c4a-ea6b-4d0b-a89b-6f42ce22b524"). InnerVolumeSpecName "kube-api-access-htk8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:22:28.600606 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.600514    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:22:28.600606 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.600577    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htk8v\" (UniqueName: \"kubernetes.io/projected/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-kube-api-access-htk8v\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:22:28.600606 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:28.600589    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:22:29.318289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:29.318243    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw" event={"ID":"7c880c4a-ea6b-4d0b-a89b-6f42ce22b524","Type":"ContainerDied","Data":"1019c5ed48e66cc8a55d7c536e7c7f152dfb6c7f6acca4c45adbb9b85f0925c3"}
Oct 31 20:22:29.318485 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:29.318328    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1019c5ed48e66cc8a55d7c536e7c7f152dfb6c7f6acca4c45adbb9b85f0925c3"
Oct 31 20:22:29.318485 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:29.318257    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"
Oct 31 20:22:36.323245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:36.323205    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:22:42.500443 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.500407    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"]
Oct 31 20:22:42.500899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.500884    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" containerName="revoke-keys"
Oct 31 20:22:42.501016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.500900    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" containerName="revoke-keys"
Oct 31 20:22:42.501016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.501014    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" containerName="revoke-keys"
Oct 31 20:22:42.504645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.504625    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.514510 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.514476    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"]
Oct 31 20:22:42.567721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.567689    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 20:22:42.567947 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.567691    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\""
Oct 31 20:22:42.568044 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.567736    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-8407e9-serving-cert\""
Oct 31 20:22:42.646583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.646533    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.646777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.646602    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55kt\" (UniqueName: \"kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.646777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.646748    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.646892 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.646805    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.666088 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.666055    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"]
Oct 31 20:22:42.670048 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.670020    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.678108 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.677802    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-16df549e-dockercfg-kzg8f\""
Oct 31 20:22:42.683599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.683566    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"]
Oct 31 20:22:42.747406 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.747377    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.747594 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.747435    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.747594 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.747482    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55kt\" (UniqueName: \"kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.747727 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.747611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.748295 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.748265    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.749847 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.749823    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.749972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.749932    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.762702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.758926    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8407e9\""
Oct 31 20:22:42.763326 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.763298    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55kt\" (UniqueName: \"kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt\") pod \"maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.822198 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.822164    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:42.849061 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.849024    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wst\" (UniqueName: \"kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst\") pod \"payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.849216 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.849147    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.950261 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.950218    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wst\" (UniqueName: \"kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst\") pod \"payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.950462 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.950312    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.951044 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.951021    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.977444 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.977412    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wst\" (UniqueName: \"kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst\") pod \"payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:42.988296 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:42.988269    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:43.104090 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.104059    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"]
Oct 31 20:22:43.105586 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:22:43.105521    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67fc953_72be_48aa_bedc_02df0bdd4695.slice/crio-cc898225fb5354cbcac36577bba81c2a4790379a16d9e86fbf744b77aacfc5c6 WatchSource:0}: Error finding container cc898225fb5354cbcac36577bba81c2a4790379a16d9e86fbf744b77aacfc5c6: Status 404 returned error can't find the container with id cc898225fb5354cbcac36577bba81c2a4790379a16d9e86fbf744b77aacfc5c6
Oct 31 20:22:43.107032 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.107014    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:22:43.280646 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.280615    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"]
Oct 31 20:22:43.281981 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:22:43.281948    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a9d2c4_771e_48f8_93e5_678fc045e45a.slice/crio-6fea8fb40f762b7ccd50c243784e353a7ffeb901da3b2fbcefaf7e603fe9831b WatchSource:0}: Error finding container 6fea8fb40f762b7ccd50c243784e353a7ffeb901da3b2fbcefaf7e603fe9831b: Status 404 returned error can't find the container with id 6fea8fb40f762b7ccd50c243784e353a7ffeb901da3b2fbcefaf7e603fe9831b
Oct 31 20:22:43.397952 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.397873    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph" event={"ID":"f67fc953-72be-48aa-bedc-02df0bdd4695","Type":"ContainerStarted","Data":"cc898225fb5354cbcac36577bba81c2a4790379a16d9e86fbf744b77aacfc5c6"}
Oct 31 20:22:43.399504 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.399478    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" event={"ID":"f4a9d2c4-771e-48f8-93e5-678fc045e45a","Type":"ContainerStarted","Data":"a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e"}
Oct 31 20:22:43.399711 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.399528    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" event={"ID":"f4a9d2c4-771e-48f8-93e5-678fc045e45a","Type":"ContainerStarted","Data":"6fea8fb40f762b7ccd50c243784e353a7ffeb901da3b2fbcefaf7e603fe9831b"}
Oct 31 20:22:43.399711 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.399601    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:43.420895 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:43.420841    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" podStartSLOduration=1.420829395 podStartE2EDuration="1.420829395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:22:43.419126716 +0000 UTC m=+3756.880501332" watchObservedRunningTime="2026-10-31 20:22:43.420829395 +0000 UTC m=+3756.882204009"
Oct 31 20:22:44.422259 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:44.422220    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph" event={"ID":"f67fc953-72be-48aa-bedc-02df0bdd4695","Type":"ContainerStarted","Data":"9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187"}
Oct 31 20:22:44.422895 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:44.422444    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:44.445080 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:44.445026    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph" podStartSLOduration=2.005198863 podStartE2EDuration="2.445010139s" totalImagesPullingTime="439.811276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:22:44.444633291 +0000 UTC m=+3757.906007905" watchObservedRunningTime="2026-10-31 20:22:44.445010139 +0000 UTC m=+3757.906384780"
Oct 31 20:22:44.899721 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:44.899683    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:22:47.045531 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:22:47.045495    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:22:50.433869 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:50.433841    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:22:54.426056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:54.426023    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:22:59.810515 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:22:59.810474    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:23:15.922729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:15.922694    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:23:17.048795 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:23:17.048756    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:23:39.825730 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:39.825690    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:23:47.047044 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:23:47.047004    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:23:49.946203 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:49.946165    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"]
Oct 31 20:23:49.952369 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:49.952324    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"]
Oct 31 20:23:49.952688 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:49.952658    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph" podUID="f67fc953-72be-48aa-bedc-02df0bdd4695" containerName="maas-api" containerID="cri-o://9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187" gracePeriod=30
Oct 31 20:23:50.254170 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.254140    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:23:50.421572 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.421503    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs\") pod \"f67fc953-72be-48aa-bedc-02df0bdd4695\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") "
Oct 31 20:23:50.421792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.421701    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt\" (UniqueName: \"kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt\") pod \"f67fc953-72be-48aa-bedc-02df0bdd4695\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") "
Oct 31 20:23:50.421792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.421756    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls\") pod \"f67fc953-72be-48aa-bedc-02df0bdd4695\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") "
Oct 31 20:23:50.421905 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.421820    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca\") pod \"f67fc953-72be-48aa-bedc-02df0bdd4695\" (UID: \"f67fc953-72be-48aa-bedc-02df0bdd4695\") "
Oct 31 20:23:50.422240 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.422213    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca" pod "f67fc953-72be-48aa-bedc-02df0bdd4695" (UID: "f67fc953-72be-48aa-bedc-02df0bdd4695"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:23:50.423800 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.423772    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs" pod "f67fc953-72be-48aa-bedc-02df0bdd4695" (UID: "f67fc953-72be-48aa-bedc-02df0bdd4695"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:23:50.424269 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.424250    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt" pod "f67fc953-72be-48aa-bedc-02df0bdd4695" (UID: "f67fc953-72be-48aa-bedc-02df0bdd4695"). InnerVolumeSpecName "kube-api-access-c55kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:23:50.424342 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.424283    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls" pod "f67fc953-72be-48aa-bedc-02df0bdd4695" (UID: "f67fc953-72be-48aa-bedc-02df0bdd4695"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:23:50.523999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.523900    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c55kt\" (UniqueName: \"kubernetes.io/projected/f67fc953-72be-48aa-bedc-02df0bdd4695-kube-api-access-c55kt\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:50.523999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.523944    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:50.523999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.523955    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67fc953-72be-48aa-bedc-02df0bdd4695-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:50.523999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.523964    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67fc953-72be-48aa-bedc-02df0bdd4695-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:23:50.841338 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.841256    5958 generic.go:356] "Generic (PLEG): container finished" podID="f67fc953-72be-48aa-bedc-02df0bdd4695" containerID="9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187" exitCode=0
Oct 31 20:23:50.841338 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.841317    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"
Oct 31 20:23:50.841530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.841337    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph" event={"ID":"f67fc953-72be-48aa-bedc-02df0bdd4695","Type":"ContainerDied","Data":"9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187"}
Oct 31 20:23:50.841530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.841394    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph" event={"ID":"f67fc953-72be-48aa-bedc-02df0bdd4695","Type":"ContainerDied","Data":"cc898225fb5354cbcac36577bba81c2a4790379a16d9e86fbf744b77aacfc5c6"}
Oct 31 20:23:50.841530 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.841415    5958 scope.go:122] "RemoveContainer" containerID="9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187"
Oct 31 20:23:50.851393 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.851370    5958 scope.go:122] "RemoveContainer" containerID="9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187"
Oct 31 20:23:50.851765 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:23:50.851745    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187\": container with ID starting with 9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187 not found: ID does not exist" containerID="9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187"
Oct 31 20:23:50.851847 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.851777    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187"} err="failed to get container status \"9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187\": rpc error: code = NotFound desc = could not find container \"9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187\": container with ID starting with 9aad3caebce899fb9e375262fbce9a5f796ad34c8dc64a62ae19af11574e6187 not found: ID does not exist"
Oct 31 20:23:50.864011 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.863982    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"]
Oct 31 20:23:50.865667 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:50.865644    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8407e9-664bd5b685-vwwph"]
Oct 31 20:23:51.799593 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:51.799527    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:23:52.662403 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:52.662367    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67fc953-72be-48aa-bedc-02df0bdd4695" path="/var/lib/kubelet/pods/f67fc953-72be-48aa-bedc-02df0bdd4695/volumes"
Oct 31 20:23:54.946720 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:23:54.946674    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerName="payload-processing" containerID="cri-o://a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e" gracePeriod=25
Oct 31 20:24:04.430234 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:04.430182    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:24:04.720252 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:04.720172    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:24:14.424099 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:14.424060    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:24:15.209699 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.209628    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:24:15.284014 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.283973    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume\") pod \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") "
Oct 31 20:24:15.284236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.284037    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst\" (UniqueName: \"kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst\") pod \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\" (UID: \"f4a9d2c4-771e-48f8-93e5-678fc045e45a\") "
Oct 31 20:24:15.284406 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.284379    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume" pod "f4a9d2c4-771e-48f8-93e5-678fc045e45a" (UID: "f4a9d2c4-771e-48f8-93e5-678fc045e45a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:24:15.286190 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.286166    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst" pod "f4a9d2c4-771e-48f8-93e5-678fc045e45a" (UID: "f4a9d2c4-771e-48f8-93e5-678fc045e45a"). InnerVolumeSpecName "kube-api-access-t7wst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:24:15.385456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.385408    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a9d2c4-771e-48f8-93e5-678fc045e45a-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:15.385456 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:15.385447    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wst\" (UniqueName: \"kubernetes.io/projected/f4a9d2c4-771e-48f8-93e5-678fc045e45a-kube-api-access-t7wst\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:16.005870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.005832    5958 generic.go:356] "Generic (PLEG): container finished" podID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerID="a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e" exitCode=0
Oct 31 20:24:16.006414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.005885    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"
Oct 31 20:24:16.006414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.005915    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" event={"ID":"f4a9d2c4-771e-48f8-93e5-678fc045e45a","Type":"ContainerDied","Data":"a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e"}
Oct 31 20:24:16.006414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.005968    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x" event={"ID":"f4a9d2c4-771e-48f8-93e5-678fc045e45a","Type":"ContainerDied","Data":"6fea8fb40f762b7ccd50c243784e353a7ffeb901da3b2fbcefaf7e603fe9831b"}
Oct 31 20:24:16.006414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.005989    5958 scope.go:122] "RemoveContainer" containerID="a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e"
Oct 31 20:24:16.016244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.016223    5958 scope.go:122] "RemoveContainer" containerID="a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e"
Oct 31 20:24:16.016594 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:24:16.016574    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e\": container with ID starting with a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e not found: ID does not exist" containerID="a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e"
Oct 31 20:24:16.016682 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.016602    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e"} err="failed to get container status \"a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e\": rpc error: code = NotFound desc = could not find container \"a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e\": container with ID starting with a72382c5337202521ac5ea3079a9aaf2baafca2eff425e61c7fa6376754c7b1e not found: ID does not exist"
Oct 31 20:24:16.031469 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.031424    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"]
Oct 31 20:24:16.036797 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.036760    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8407e9-7b7d8679c2rs6x"]
Oct 31 20:24:16.670461 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:16.670425    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" path="/var/lib/kubelet/pods/f4a9d2c4-771e-48f8-93e5-678fc045e45a/volumes"
Oct 31 20:24:17.044922 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:24:17.044827    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:24:19.992028 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:19.991993    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:24:35.738597 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.738538    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"]
Oct 31 20:24:35.739059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.739044    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f67fc953-72be-48aa-bedc-02df0bdd4695" containerName="maas-api"
Oct 31 20:24:35.739127 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.739060    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67fc953-72be-48aa-bedc-02df0bdd4695" containerName="maas-api"
Oct 31 20:24:35.739127 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.739079    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerName="payload-processing"
Oct 31 20:24:35.739127 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.739085    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerName="payload-processing"
Oct 31 20:24:35.739229 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.739163    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67fc953-72be-48aa-bedc-02df0bdd4695" containerName="maas-api"
Oct 31 20:24:35.739229 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.739173    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a9d2c4-771e-48f8-93e5-678fc045e45a" containerName="payload-processing"
Oct 31 20:24:35.744026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.744004    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.751698 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.751628    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"]
Oct 31 20:24:35.799804 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.799776    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:24:35.799999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.799931    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:24:35.824063 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.824026    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.824255 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.824091    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpwkd\" (UniqueName: \"kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.824255 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.824163    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.925709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.925667    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.925915 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.925735    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpwkd\" (UniqueName: \"kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.925915 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.925754    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.926433 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.926411    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.928191 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.928173    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:35.939278 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:35.939253    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpwkd\" (UniqueName: \"kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd\") pod \"maas-api-revoke-sub-452b6d7b-hw8bm\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:36.060410 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:36.060324    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:36.296614 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:36.296565    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:24:36.338056 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:36.338030    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"]
Oct 31 20:24:36.340275 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:24:36.340247    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50304b0_7a3a_4c95_a245_bcb5e2b95e6d.slice/crio-fe1d4a3a27ad849920da076aa592ce641f04d584010e6d1318ef4affc6c42382 WatchSource:0}: Error finding container fe1d4a3a27ad849920da076aa592ce641f04d584010e6d1318ef4affc6c42382: Status 404 returned error can't find the container with id fe1d4a3a27ad849920da076aa592ce641f04d584010e6d1318ef4affc6c42382
Oct 31 20:24:37.172251 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:37.172207    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm" event={"ID":"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d","Type":"ContainerStarted","Data":"d86c655b7d83b13082fca8ca87e643e6271f94e5a925ab4add76cf795de8f90f"}
Oct 31 20:24:37.172802 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:37.172314    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm" event={"ID":"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d","Type":"ContainerStarted","Data":"fe1d4a3a27ad849920da076aa592ce641f04d584010e6d1318ef4affc6c42382"}
Oct 31 20:24:37.196428 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:37.196361    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm" podStartSLOduration=2.196343826 podStartE2EDuration="2.196343826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:24:37.192304202 +0000 UTC m=+3870.653678817" watchObservedRunningTime="2026-10-31 20:24:37.196343826 +0000 UTC m=+3870.657718442"
Oct 31 20:24:38.183316 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:38.183253    5958 generic.go:356] "Generic (PLEG): container finished" podID="c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" containerID="d86c655b7d83b13082fca8ca87e643e6271f94e5a925ab4add76cf795de8f90f" exitCode=0
Oct 31 20:24:38.183860 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:38.183335    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm" event={"ID":"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d","Type":"ContainerDied","Data":"d86c655b7d83b13082fca8ca87e643e6271f94e5a925ab4add76cf795de8f90f"}
Oct 31 20:24:39.579824 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.579789    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:39.597175 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.597132    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca\") pod \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") "
Oct 31 20:24:39.597436 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.597422    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca\") pod \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") "
Oct 31 20:24:39.597528 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.597503    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca" pod "c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" (UID: "c50304b0-7a3a-4c95-a245-bcb5e2b95e6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:24:39.597709 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.597530    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd\" (UniqueName: \"kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd\") pod \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\" (UID: \"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d\") "
Oct 31 20:24:39.597997 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.597972    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:39.599723 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.599701    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca" pod "c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" (UID: "c50304b0-7a3a-4c95-a245-bcb5e2b95e6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:24:39.600870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.600850    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd" pod "c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" (UID: "c50304b0-7a3a-4c95-a245-bcb5e2b95e6d"). InnerVolumeSpecName "kube-api-access-hpwkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:24:39.699297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.699253    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpwkd\" (UniqueName: \"kubernetes.io/projected/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-kube-api-access-hpwkd\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:39.699297 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:39.699295    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:40.194391 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:40.194361    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"
Oct 31 20:24:40.194391 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:40.194373    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm" event={"ID":"c50304b0-7a3a-4c95-a245-bcb5e2b95e6d","Type":"ContainerDied","Data":"fe1d4a3a27ad849920da076aa592ce641f04d584010e6d1318ef4affc6c42382"}
Oct 31 20:24:40.194636 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:40.194426    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe1d4a3a27ad849920da076aa592ce641f04d584010e6d1318ef4affc6c42382"
Oct 31 20:24:47.045494 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:24:47.045409    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:24:48.001296 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:48.001257    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:24:51.948966 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.948920    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"]
Oct 31 20:24:51.949863 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.949764    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" containerName="revoke-keys"
Oct 31 20:24:51.949863 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.949784    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" containerName="revoke-keys"
Oct 31 20:24:51.950051 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.949890    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" containerName="revoke-keys"
Oct 31 20:24:51.953759 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.953733    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:51.961219 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.961177    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:24:51.961669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.961638    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:24:51.964630 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:51.964589    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"]
Oct 31 20:24:52.096276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.096232    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.096499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.096360    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx47c\" (UniqueName: \"kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.096499 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.096394    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.197206 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.197160    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.197439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.197237    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx47c\" (UniqueName: \"kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.197439 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.197275    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.198144 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.198114    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.199976 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.199913    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.211426 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.211387    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx47c\" (UniqueName: \"kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c\") pod \"maas-api-revoke-sub-3e31a506-mgcvq\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.278801 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.278758    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:52.765617 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:52.732032    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"]
Oct 31 20:24:53.055567 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:53.055105    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"]
Oct 31 20:24:53.074955 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:53.074599    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3384e0-mlwj6"]
Oct 31 20:24:53.077663 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:53.077637    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:24:53.358632 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:53.353568    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq" event={"ID":"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6","Type":"ContainerStarted","Data":"6deff65dad2df46f1f9d058382a0a4e6959c6adf3145c44c9a46bcb68d682f0f"}
Oct 31 20:24:53.358632 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:53.353639    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq" event={"ID":"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6","Type":"ContainerStarted","Data":"ebc432986c28bcc184ce4ad9a0329cf98ef0b6f0f7b4ff5670a787fd0e6b3b10"}
Oct 31 20:24:53.396165 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:53.396096    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq" podStartSLOduration=2.396082986 podStartE2EDuration="2.396082986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:24:53.395862847 +0000 UTC m=+3886.857237566" watchObservedRunningTime="2026-10-31 20:24:53.396082986 +0000 UTC m=+3886.857457601"
Oct 31 20:24:54.360920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:54.360822    5958 generic.go:356] "Generic (PLEG): container finished" podID="2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" containerID="6deff65dad2df46f1f9d058382a0a4e6959c6adf3145c44c9a46bcb68d682f0f" exitCode=0
Oct 31 20:24:54.360920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:54.360865    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq" event={"ID":"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6","Type":"ContainerDied","Data":"6deff65dad2df46f1f9d058382a0a4e6959c6adf3145c44c9a46bcb68d682f0f"}
Oct 31 20:24:54.662897 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:54.662855    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453b70ce-ca12-4975-af54-0ef5f6026c9b" path="/var/lib/kubelet/pods/453b70ce-ca12-4975-af54-0ef5f6026c9b/volumes"
Oct 31 20:24:55.508666 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.508637    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:55.662136 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.662076    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c\" (UniqueName: \"kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c\") pod \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") "
Oct 31 20:24:55.662442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.662225    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca\") pod \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") "
Oct 31 20:24:55.662442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.662267    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca\") pod \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\" (UID: \"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6\") "
Oct 31 20:24:55.662702 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.662674    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca" pod "2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" (UID: "2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:24:55.664613 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.664556    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c" pod "2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" (UID: "2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6"). InnerVolumeSpecName "kube-api-access-wx47c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:24:55.664731 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.664675    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca" pod "2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" (UID: "2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:24:55.763488 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.763450    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:55.763488 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.763483    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:55.763488 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:55.763493    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx47c\" (UniqueName: \"kubernetes.io/projected/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6-kube-api-access-wx47c\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:56.373599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:56.373565    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq" event={"ID":"2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6","Type":"ContainerDied","Data":"ebc432986c28bcc184ce4ad9a0329cf98ef0b6f0f7b4ff5670a787fd0e6b3b10"}
Oct 31 20:24:56.373819 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:56.373609    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"
Oct 31 20:24:56.373819 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:24:56.373614    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc432986c28bcc184ce4ad9a0329cf98ef0b6f0f7b4ff5670a787fd0e6b3b10"
Oct 31 20:25:02.826362 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:02.826308    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:25:07.318263 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:07.318223    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:25:10.588012 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:10.587977    5958 scope.go:122] "RemoveContainer" containerID="a7c7cb814beb654eb0ef116e05bacd238af52384da7c216cf716fb49523065a5"
Oct 31 20:25:12.493369 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.493334    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"]
Oct 31 20:25:12.493852 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.493836    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" containerName="revoke-keys"
Oct 31 20:25:12.493899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.493854    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" containerName="revoke-keys"
Oct 31 20:25:12.493959 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.493950    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" containerName="revoke-keys"
Oct 31 20:25:12.497177 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.497161    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.504821 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.504800    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:25:12.506485 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.506453    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:25:12.520941 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.520643    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.520941 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.520748    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wqqm\" (UniqueName: \"kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.520941 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.520781    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.522765 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.522736    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"]
Oct 31 20:25:12.622184 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.622141    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.622445 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.622228    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wqqm\" (UniqueName: \"kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.622445 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.622262    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.623045 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.623026    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.624582 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.624561    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.636186 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.636155    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wqqm\" (UniqueName: \"kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm\") pod \"maas-api-revoke-sub-df8716d9-ft4sx\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:12.810413 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:12.810323    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:13.087424 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:13.087394    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"]
Oct 31 20:25:13.088162 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:25:13.088122    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7080b189_6fdd_4106_8daa_7f52d6ad8b74.slice/crio-279f548cc3d64d9820e526631ed97299932ec4dfbabe23442598641b9bfc2d49 WatchSource:0}: Error finding container 279f548cc3d64d9820e526631ed97299932ec4dfbabe23442598641b9bfc2d49: Status 404 returned error can't find the container with id 279f548cc3d64d9820e526631ed97299932ec4dfbabe23442598641b9bfc2d49
Oct 31 20:25:13.537981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:13.537947    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx" event={"ID":"7080b189-6fdd-4106-8daa-7f52d6ad8b74","Type":"ContainerStarted","Data":"5236f3bdbb235c5a13ad4de7f7bc3f127f9bf9ff1989dfdfcd33cc9a83a43b29"}
Oct 31 20:25:13.538422 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:13.538004    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx" event={"ID":"7080b189-6fdd-4106-8daa-7f52d6ad8b74","Type":"ContainerStarted","Data":"279f548cc3d64d9820e526631ed97299932ec4dfbabe23442598641b9bfc2d49"}
Oct 31 20:25:13.559610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:13.559556    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx" podStartSLOduration=1.559523418 podStartE2EDuration="1.559523418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:13.558626509 +0000 UTC m=+3907.020001125" watchObservedRunningTime="2026-10-31 20:25:13.559523418 +0000 UTC m=+3907.020898035"
Oct 31 20:25:14.545758 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:14.545724    5958 generic.go:356] "Generic (PLEG): container finished" podID="7080b189-6fdd-4106-8daa-7f52d6ad8b74" containerID="5236f3bdbb235c5a13ad4de7f7bc3f127f9bf9ff1989dfdfcd33cc9a83a43b29" exitCode=0
Oct 31 20:25:14.546172 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:14.545807    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx" event={"ID":"7080b189-6fdd-4106-8daa-7f52d6ad8b74","Type":"ContainerDied","Data":"5236f3bdbb235c5a13ad4de7f7bc3f127f9bf9ff1989dfdfcd33cc9a83a43b29"}
Oct 31 20:25:15.692026 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.691999    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:15.751870 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.751838    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm\" (UniqueName: \"kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm\") pod \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") "
Oct 31 20:25:15.752064 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.751918    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca\") pod \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") "
Oct 31 20:25:15.752064 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.752036    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca\") pod \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\" (UID: \"7080b189-6fdd-4106-8daa-7f52d6ad8b74\") "
Oct 31 20:25:15.752318 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.752293    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca" pod "7080b189-6fdd-4106-8daa-7f52d6ad8b74" (UID: "7080b189-6fdd-4106-8daa-7f52d6ad8b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:25:15.752402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.752373    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-openshift-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:15.754036 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.754010    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca" pod "7080b189-6fdd-4106-8daa-7f52d6ad8b74" (UID: "7080b189-6fdd-4106-8daa-7f52d6ad8b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:15.754127 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.754082    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm" pod "7080b189-6fdd-4106-8daa-7f52d6ad8b74" (UID: "7080b189-6fdd-4106-8daa-7f52d6ad8b74"). InnerVolumeSpecName "kube-api-access-2wqqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:25:15.853268 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.853180    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7080b189-6fdd-4106-8daa-7f52d6ad8b74-maas-api-xks-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:15.853268 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:15.853216    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wqqm\" (UniqueName: \"kubernetes.io/projected/7080b189-6fdd-4106-8daa-7f52d6ad8b74-kube-api-access-2wqqm\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:16.558206 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:16.558175    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"
Oct 31 20:25:16.558376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:16.558176    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx" event={"ID":"7080b189-6fdd-4106-8daa-7f52d6ad8b74","Type":"ContainerDied","Data":"279f548cc3d64d9820e526631ed97299932ec4dfbabe23442598641b9bfc2d49"}
Oct 31 20:25:16.558376 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:16.558310    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="279f548cc3d64d9820e526631ed97299932ec4dfbabe23442598641b9bfc2d49"
Oct 31 20:25:17.044154 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:25:17.044111    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:25:23.386059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:23.386022    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:25:24.038068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.038030    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"]
Oct 31 20:25:24.047009 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.046977    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9zt2n"]
Oct 31 20:25:24.662886 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.662851    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbdbfee-7809-4798-bf05-d95dd3ee166a" path="/var/lib/kubelet/pods/3dbdbfee-7809-4798-bf05-d95dd3ee166a/volumes"
Oct 31 20:25:24.796593 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.796555    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"]
Oct 31 20:25:24.797074 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.797059    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7080b189-6fdd-4106-8daa-7f52d6ad8b74" containerName="revoke-keys"
Oct 31 20:25:24.797129 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.797076    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7080b189-6fdd-4106-8daa-7f52d6ad8b74" containerName="revoke-keys"
Oct 31 20:25:24.797166 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.797157    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7080b189-6fdd-4106-8daa-7f52d6ad8b74" containerName="revoke-keys"
Oct 31 20:25:24.801797 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.801768    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.810383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.810353    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 20:25:24.811085 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.810438    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 20:25:24.813179 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.813153    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\""
Oct 31 20:25:24.815875 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.815849    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"]
Oct 31 20:25:24.822740 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.822431    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"]
Oct 31 20:25:24.843833 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.843798    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:24.844002 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.843807    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"]
Oct 31 20:25:24.851693 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.851662    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.851845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.851725    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.851845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.851748    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.851845 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.851791    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfzw\" (UniqueName: \"kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.952934 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.952836    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/560513c8-27f1-49d1-931d-c6a7c55fe344-maas-api-tls\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:24.953097 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.952937    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsj9\" (UniqueName: \"kubernetes.io/projected/560513c8-27f1-49d1-931d-c6a7c55fe344-kube-api-access-fbsj9\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:24.953097 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.952990    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.953097 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.953012    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.953097 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.953090    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfzw\" (UniqueName: \"kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.953247 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.953145    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/560513c8-27f1-49d1-931d-c6a7c55fe344-metrics-certs\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:24.953247 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.953236    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/560513c8-27f1-49d1-931d-c6a7c55fe344-openshift-service-ca\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:24.953348 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.953272    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.953750 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.953729    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.955650 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.955627    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.955776 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.955757    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:24.967256 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:24.967229    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfzw\" (UniqueName: \"kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw\") pod \"maas-api-7b784f56d7-tcz2z\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:25.054922 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.054882    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/560513c8-27f1-49d1-931d-c6a7c55fe344-metrics-certs\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.055151 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.054951    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/560513c8-27f1-49d1-931d-c6a7c55fe344-openshift-service-ca\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.055151 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.054985    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/560513c8-27f1-49d1-931d-c6a7c55fe344-maas-api-tls\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.055151 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.055001    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsj9\" (UniqueName: \"kubernetes.io/projected/560513c8-27f1-49d1-931d-c6a7c55fe344-kube-api-access-fbsj9\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.055804 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.055777    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/560513c8-27f1-49d1-931d-c6a7c55fe344-openshift-service-ca\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.057792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.057768    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/560513c8-27f1-49d1-931d-c6a7c55fe344-maas-api-tls\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.057917 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.057894    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/560513c8-27f1-49d1-931d-c6a7c55fe344-metrics-certs\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.086836 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.086806    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsj9\" (UniqueName: \"kubernetes.io/projected/560513c8-27f1-49d1-931d-c6a7c55fe344-kube-api-access-fbsj9\") pod \"maas-api-7fd9fcc7c9-gpftj\" (UID: \"560513c8-27f1-49d1-931d-c6a7c55fe344\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.142364 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.142327    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:25.157937 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.157905    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:25.436639 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.436612    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"]
Oct 31 20:25:25.438064 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:25:25.438037    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b093a2d_3279_42d8_8292_2cba007dac1e.slice/crio-c19e828c1c81399c3fddfe6966331af066ef1139b3845d0bb4cc929a60ec48bb WatchSource:0}: Error finding container c19e828c1c81399c3fddfe6966331af066ef1139b3845d0bb4cc929a60ec48bb: Status 404 returned error can't find the container with id c19e828c1c81399c3fddfe6966331af066ef1139b3845d0bb4cc929a60ec48bb
Oct 31 20:25:25.450962 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.450936    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"]
Oct 31 20:25:25.452962 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:25:25.452936    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560513c8_27f1_49d1_931d_c6a7c55fe344.slice/crio-5de72c5090deae5344ce46a278b4dc81aa77a298d29bebe821a168780089bcc1 WatchSource:0}: Error finding container 5de72c5090deae5344ce46a278b4dc81aa77a298d29bebe821a168780089bcc1: Status 404 returned error can't find the container with id 5de72c5090deae5344ce46a278b4dc81aa77a298d29bebe821a168780089bcc1
Oct 31 20:25:25.612327 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.612242    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z" event={"ID":"1b093a2d-3279-42d8-8292-2cba007dac1e","Type":"ContainerStarted","Data":"c19e828c1c81399c3fddfe6966331af066ef1139b3845d0bb4cc929a60ec48bb"}
Oct 31 20:25:25.613748 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:25.613716    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj" event={"ID":"560513c8-27f1-49d1-931d-c6a7c55fe344","Type":"ContainerStarted","Data":"5de72c5090deae5344ce46a278b4dc81aa77a298d29bebe821a168780089bcc1"}
Oct 31 20:25:26.620900 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:26.620861    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z" event={"ID":"1b093a2d-3279-42d8-8292-2cba007dac1e","Type":"ContainerStarted","Data":"98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f"}
Oct 31 20:25:26.621382 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:26.620926    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:26.622527 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:26.622503    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj" event={"ID":"560513c8-27f1-49d1-931d-c6a7c55fe344","Type":"ContainerStarted","Data":"6964819d36c6bd053d8f64286618549c2b4244bbbc38d8035634b93f1f6fdcb1"}
Oct 31 20:25:26.622678 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:26.622635    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:26.647835 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:26.647776    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z" podStartSLOduration=2.139392806 podStartE2EDuration="2.647761813s" totalImagesPullingTime="508.369007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:26.64650939 +0000 UTC m=+3920.107884009" watchObservedRunningTime="2026-10-31 20:25:26.647761813 +0000 UTC m=+3920.109136437"
Oct 31 20:25:26.672249 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:26.672191    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj" podStartSLOduration=2.210993475 podStartE2EDuration="2.67217858s" totalImagesPullingTime="461.185105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:26.670480777 +0000 UTC m=+3920.131855393" watchObservedRunningTime="2026-10-31 20:25:26.67217858 +0000 UTC m=+3920.133553194"
Oct 31 20:25:32.634046 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:32.634015    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:32.634521 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:32.634411    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-gpftj"
Oct 31 20:25:32.720061 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:32.720028    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"]
Oct 31 20:25:32.720274 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:32.720253    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z" podUID="1b093a2d-3279-42d8-8292-2cba007dac1e" containerName="maas-api" containerID="cri-o://98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f" gracePeriod=30
Oct 31 20:25:32.988618 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:32.988590    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:33.142040 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.142005    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs\") pod \"1b093a2d-3279-42d8-8292-2cba007dac1e\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") "
Oct 31 20:25:33.142218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.142094    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca\") pod \"1b093a2d-3279-42d8-8292-2cba007dac1e\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") "
Oct 31 20:25:33.142218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.142142    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls\") pod \"1b093a2d-3279-42d8-8292-2cba007dac1e\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") "
Oct 31 20:25:33.142218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.142193    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw\" (UniqueName: \"kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw\") pod \"1b093a2d-3279-42d8-8292-2cba007dac1e\" (UID: \"1b093a2d-3279-42d8-8292-2cba007dac1e\") "
Oct 31 20:25:33.142484 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.142459    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca" pod "1b093a2d-3279-42d8-8292-2cba007dac1e" (UID: "1b093a2d-3279-42d8-8292-2cba007dac1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:25:33.144521 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.144459    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs" pod "1b093a2d-3279-42d8-8292-2cba007dac1e" (UID: "1b093a2d-3279-42d8-8292-2cba007dac1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:33.144676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.144533    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls" pod "1b093a2d-3279-42d8-8292-2cba007dac1e" (UID: "1b093a2d-3279-42d8-8292-2cba007dac1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:33.144676 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.144666    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw" pod "1b093a2d-3279-42d8-8292-2cba007dac1e" (UID: "1b093a2d-3279-42d8-8292-2cba007dac1e"). InnerVolumeSpecName "kube-api-access-rvfzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:25:33.243659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.243622    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfzw\" (UniqueName: \"kubernetes.io/projected/1b093a2d-3279-42d8-8292-2cba007dac1e-kube-api-access-rvfzw\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:33.243659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.243655    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-metrics-certs\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:33.243659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.243665    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b093a2d-3279-42d8-8292-2cba007dac1e-openshift-service-ca\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:33.243998 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.243675    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b093a2d-3279-42d8-8292-2cba007dac1e-maas-api-tls\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:33.662823 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.662782    5958 generic.go:356] "Generic (PLEG): container finished" podID="1b093a2d-3279-42d8-8292-2cba007dac1e" containerID="98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f" exitCode=0
Oct 31 20:25:33.663272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.662842    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"
Oct 31 20:25:33.663272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.662883    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z" event={"ID":"1b093a2d-3279-42d8-8292-2cba007dac1e","Type":"ContainerDied","Data":"98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f"}
Oct 31 20:25:33.663272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.662949    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z" event={"ID":"1b093a2d-3279-42d8-8292-2cba007dac1e","Type":"ContainerDied","Data":"c19e828c1c81399c3fddfe6966331af066ef1139b3845d0bb4cc929a60ec48bb"}
Oct 31 20:25:33.663272 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.662984    5958 scope.go:122] "RemoveContainer" containerID="98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f"
Oct 31 20:25:33.676337 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.676323    5958 scope.go:122] "RemoveContainer" containerID="98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f"
Oct 31 20:25:33.676609 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:25:33.676579    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f\": container with ID starting with 98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f not found: ID does not exist" containerID="98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f"
Oct 31 20:25:33.676728 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.676615    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f"} err="failed to get container status \"98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f\": rpc error: code = NotFound desc = could not find container \"98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f\": container with ID starting with 98dfaa4bb8862f9cf070cd605b5055ef650f06e3ccdc02ca499360d5e1f5e53f not found: ID does not exist"
Oct 31 20:25:33.689608 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.689587    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"]
Oct 31 20:25:33.695121 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:33.695098    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-tcz2z"]
Oct 31 20:25:34.662244 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:34.662197    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b093a2d-3279-42d8-8292-2cba007dac1e" path="/var/lib/kubelet/pods/1b093a2d-3279-42d8-8292-2cba007dac1e/volumes"
Oct 31 20:25:36.276261 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.276224    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"]
Oct 31 20:25:36.276757 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.276741    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1b093a2d-3279-42d8-8292-2cba007dac1e" containerName="maas-api"
Oct 31 20:25:36.276806 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.276758    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b093a2d-3279-42d8-8292-2cba007dac1e" containerName="maas-api"
Oct 31 20:25:36.276843 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.276828    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b093a2d-3279-42d8-8292-2cba007dac1e" containerName="maas-api"
Oct 31 20:25:36.281199 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.281179    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.300139 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.300104    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4554a17c-dockercfg-mp8hz\""
Oct 31 20:25:36.301865 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.300512    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4554a17c\""
Oct 31 20:25:36.302028 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.300849    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"]
Oct 31 20:25:36.382235 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.382201    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdhb\" (UniqueName: \"kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb\") pod \"payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") " pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.382467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.382323    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume\") pod \"payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") " pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.483401 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.483357    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdhb\" (UniqueName: \"kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb\") pod \"payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") " pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.483665 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.483467    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume\") pod \"payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") " pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.484407 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.484377    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume\") pod \"payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") " pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.501175 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.501140    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdhb\" (UniqueName: \"kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb\") pod \"payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") " pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.605777 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.605689    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:36.916092 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:36.916052    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"]
Oct 31 20:25:36.919289 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:25:36.919256    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b10c23_c3ef_4962_b6b2_879d4110ebde.slice/crio-834e18c48b67c75b195f8f96ac3619fd4dc22f43f556d9b5089aba1ea1349e35 WatchSource:0}: Error finding container 834e18c48b67c75b195f8f96ac3619fd4dc22f43f556d9b5089aba1ea1349e35: Status 404 returned error can't find the container with id 834e18c48b67c75b195f8f96ac3619fd4dc22f43f556d9b5089aba1ea1349e35
Oct 31 20:25:37.690915 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:37.690878    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l" event={"ID":"59b10c23-c3ef-4962-b6b2-879d4110ebde","Type":"ContainerStarted","Data":"2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68"}
Oct 31 20:25:37.691345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:37.690935    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l" event={"ID":"59b10c23-c3ef-4962-b6b2-879d4110ebde","Type":"ContainerStarted","Data":"834e18c48b67c75b195f8f96ac3619fd4dc22f43f556d9b5089aba1ea1349e35"}
Oct 31 20:25:37.691345 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:37.691086    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:25:37.715309 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:37.714991    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l" podStartSLOduration=1.714971676 podStartE2EDuration="1.714971676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:37.71458723 +0000 UTC m=+3931.175961846" watchObservedRunningTime="2026-10-31 20:25:37.714971676 +0000 UTC m=+3931.176346291"
Oct 31 20:25:38.824956 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:38.824918    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:25:47.045065 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:25:47.045025    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:25:48.700467 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:25:48.700435    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:26:00.918148 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:00.918113    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"]
Oct 31 20:26:03.636281 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:03.636244    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:26:05.918611 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:05.918524    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l" podUID="59b10c23-c3ef-4962-b6b2-879d4110ebde" containerName="payload-processing" containerID="cri-o://2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68" gracePeriod=25
Oct 31 20:26:06.189729 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.189701    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:26:06.310711 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.310676    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume\") pod \"59b10c23-c3ef-4962-b6b2-879d4110ebde\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") "
Oct 31 20:26:06.310894 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.310730    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb\" (UniqueName: \"kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb\") pod \"59b10c23-c3ef-4962-b6b2-879d4110ebde\" (UID: \"59b10c23-c3ef-4962-b6b2-879d4110ebde\") "
Oct 31 20:26:06.311085 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.311064    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume" pod "59b10c23-c3ef-4962-b6b2-879d4110ebde" (UID: "59b10c23-c3ef-4962-b6b2-879d4110ebde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:26:06.312806 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.312787    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb" pod "59b10c23-c3ef-4962-b6b2-879d4110ebde" (UID: "59b10c23-c3ef-4962-b6b2-879d4110ebde"). InnerVolumeSpecName "kube-api-access-dfdhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:26:06.411570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.411504    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59b10c23-c3ef-4962-b6b2-879d4110ebde-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:06.411570 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.411537    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdhb\" (UniqueName: \"kubernetes.io/projected/59b10c23-c3ef-4962-b6b2-879d4110ebde-kube-api-access-dfdhb\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:06.857276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.857181    5958 generic.go:356] "Generic (PLEG): container finished" podID="59b10c23-c3ef-4962-b6b2-879d4110ebde" containerID="2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68" exitCode=0
Oct 31 20:26:06.857276 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.857225    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l" event={"ID":"59b10c23-c3ef-4962-b6b2-879d4110ebde","Type":"ContainerDied","Data":"2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68"}
Oct 31 20:26:06.857505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.857299    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l" event={"ID":"59b10c23-c3ef-4962-b6b2-879d4110ebde","Type":"ContainerDied","Data":"834e18c48b67c75b195f8f96ac3619fd4dc22f43f556d9b5089aba1ea1349e35"}
Oct 31 20:26:06.857505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.857235    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"
Oct 31 20:26:06.857505 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.857333    5958 scope.go:122] "RemoveContainer" containerID="2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68"
Oct 31 20:26:06.866868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.866846    5958 scope.go:122] "RemoveContainer" containerID="2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68"
Oct 31 20:26:06.867197 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:26:06.867178    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68\": container with ID starting with 2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68 not found: ID does not exist" containerID="2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68"
Oct 31 20:26:06.867270 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.867207    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68"} err="failed to get container status \"2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68\": rpc error: code = NotFound desc = could not find container \"2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68\": container with ID starting with 2af445f7642a64388b0f0a55a8de47e3fcc18a64eeb00a2fdb787b32cabd6e68 not found: ID does not exist"
Oct 31 20:26:06.878158 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.878129    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"]
Oct 31 20:26:06.882973 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:06.882947    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4554a17c-bfdd8df5-89k9l"]
Oct 31 20:26:08.662231 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:08.662194    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b10c23-c3ef-4962-b6b2-879d4110ebde" path="/var/lib/kubelet/pods/59b10c23-c3ef-4962-b6b2-879d4110ebde/volumes"
Oct 31 20:26:10.619627 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:10.619592    5958 scope.go:122] "RemoveContainer" containerID="1b790c7cca8dff5dc303a5b7b558e64ac016e888fc226c6a925ba3f3b4df545a"
Oct 31 20:26:10.631170 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:10.631147    5958 scope.go:122] "RemoveContainer" containerID="b4cf6ad1658687fc17745436c039f23fa9814165dc06e63b9c11a250d7ccffa0"
Oct 31 20:26:14.135240 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.135200    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"]
Oct 31 20:26:14.135806 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.135786    5958 cpu_manager.go:414] "containerMap: removing container" podUID="59b10c23-c3ef-4962-b6b2-879d4110ebde" containerName="payload-processing"
Oct 31 20:26:14.135854 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.135807    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b10c23-c3ef-4962-b6b2-879d4110ebde" containerName="payload-processing"
Oct 31 20:26:14.135899 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.135892    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b10c23-c3ef-4962-b6b2-879d4110ebde" containerName="payload-processing"
Oct 31 20:26:14.139348 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.139320    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.149822 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.149798    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-33c6d1\""
Oct 31 20:26:14.167367 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.167320    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"]
Oct 31 20:26:14.294642 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.294600    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.295006 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.294707    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxls7\" (UniqueName: \"kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7\") pod \"payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.395666 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.395576    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxls7\" (UniqueName: \"kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7\") pod \"payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.395829 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.395693    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.396352 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.396334    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.426536 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.426497    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxls7\" (UniqueName: \"kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7\") pod \"payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.453027 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.452723    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.738624 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.738598    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"]
Oct 31 20:26:14.740856 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:26:14.740828    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934df694_d4e6_4820_8a57_59cf204c6c27.slice/crio-66fb519038c8023daafaf0fa19841b2d63f6513da5623ac230d36b4099b07ed6 WatchSource:0}: Error finding container 66fb519038c8023daafaf0fa19841b2d63f6513da5623ac230d36b4099b07ed6: Status 404 returned error can't find the container with id 66fb519038c8023daafaf0fa19841b2d63f6513da5623ac230d36b4099b07ed6
Oct 31 20:26:14.907327 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.907291    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx" event={"ID":"934df694-d4e6-4820-8a57-59cf204c6c27","Type":"ContainerStarted","Data":"0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98"}
Oct 31 20:26:14.907535 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.907344    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx" event={"ID":"934df694-d4e6-4820-8a57-59cf204c6c27","Type":"ContainerStarted","Data":"66fb519038c8023daafaf0fa19841b2d63f6513da5623ac230d36b4099b07ed6"}
Oct 31 20:26:14.907535 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.907467    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:14.934230 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:14.934176    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx" podStartSLOduration=0.934160912 podStartE2EDuration="934.160912ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:14.931856436 +0000 UTC m=+3968.393231072" watchObservedRunningTime="2026-10-31 20:26:14.934160912 +0000 UTC m=+3968.395535526"
Oct 31 20:26:16.613694 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:16.613611    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:26:17.045251 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:26:17.045209    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:26:25.917318 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:26:25.917285    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:26:47.045633 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:26:47.045589    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:27:10.027391 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:10.027348    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"]
Oct 31 20:27:10.030211 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:10.030181    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6slj"]
Oct 31 20:27:10.662831 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:10.662797    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98c0ef0-03d4-4b05-90ad-d794057f06bb" path="/var/lib/kubelet/pods/b98c0ef0-03d4-4b05-90ad-d794057f06bb/volumes"
Oct 31 20:27:10.664454 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:10.664433    5958 scope.go:122] "RemoveContainer" containerID="44a3922c8325d170afac627d4ddca4b9858bcf81e0007120d5ba540dc1884d51"
Oct 31 20:27:17.045099 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:27:17.045057    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:27:29.026898 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:29.026859    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"]
Oct 31 20:27:29.030946 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:29.030916    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ldvjw"]
Oct 31 20:27:30.663022 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:30.662989    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c880c4a-ea6b-4d0b-a89b-6f42ce22b524" path="/var/lib/kubelet/pods/7c880c4a-ea6b-4d0b-a89b-6f42ce22b524/volumes"
Oct 31 20:27:47.045137 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:27:47.045051    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:27:58.175956 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:27:58.175917    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"]
Oct 31 20:28:03.176558 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.176476    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx" podUID="934df694-d4e6-4820-8a57-59cf204c6c27" containerName="payload-processing" containerID="cri-o://0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98" gracePeriod=25
Oct 31 20:28:03.438114 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.438048    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:28:03.538189 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.538153    5958 generic.go:356] "Generic (PLEG): container finished" podID="934df694-d4e6-4820-8a57-59cf204c6c27" containerID="0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98" exitCode=0
Oct 31 20:28:03.538364 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.538197    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"
Oct 31 20:28:03.538364 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.538201    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx" event={"ID":"934df694-d4e6-4820-8a57-59cf204c6c27","Type":"ContainerDied","Data":"0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98"}
Oct 31 20:28:03.538364 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.538305    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx" event={"ID":"934df694-d4e6-4820-8a57-59cf204c6c27","Type":"ContainerDied","Data":"66fb519038c8023daafaf0fa19841b2d63f6513da5623ac230d36b4099b07ed6"}
Oct 31 20:28:03.538364 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.538333    5958 scope.go:122] "RemoveContainer" containerID="0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98"
Oct 31 20:28:03.547412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.547390    5958 scope.go:122] "RemoveContainer" containerID="0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98"
Oct 31 20:28:03.547740 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:28:03.547716    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98\": container with ID starting with 0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98 not found: ID does not exist" containerID="0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98"
Oct 31 20:28:03.547812 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.547754    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98"} err="failed to get container status \"0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98\": rpc error: code = NotFound desc = could not find container \"0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98\": container with ID starting with 0ba2a085c9df81818d2bc27f65cb7bab7b264066668dc30277488d726f99fa98 not found: ID does not exist"
Oct 31 20:28:03.589440 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.589401    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7\" (UniqueName: \"kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7\") pod \"934df694-d4e6-4820-8a57-59cf204c6c27\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") "
Oct 31 20:28:03.589654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.589467    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume\") pod \"934df694-d4e6-4820-8a57-59cf204c6c27\" (UID: \"934df694-d4e6-4820-8a57-59cf204c6c27\") "
Oct 31 20:28:03.589884 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.589862    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume" pod "934df694-d4e6-4820-8a57-59cf204c6c27" (UID: "934df694-d4e6-4820-8a57-59cf204c6c27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:03.591645 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.591619    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7" pod "934df694-d4e6-4820-8a57-59cf204c6c27" (UID: "934df694-d4e6-4820-8a57-59cf204c6c27"). InnerVolumeSpecName "kube-api-access-dxls7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:03.691442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.691337    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxls7\" (UniqueName: \"kubernetes.io/projected/934df694-d4e6-4820-8a57-59cf204c6c27-kube-api-access-dxls7\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:03.691442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.691379    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/934df694-d4e6-4820-8a57-59cf204c6c27-plugins-config-volume\") on node \"ip-10-0-138-35.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:03.862279 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.862239    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"]
Oct 31 20:28:03.867414 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:03.867385    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-33c6d1-7b58bf8887-9hndx"]
Oct 31 20:28:04.662706 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:04.662674    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934df694-d4e6-4820-8a57-59cf204c6c27" path="/var/lib/kubelet/pods/934df694-d4e6-4820-8a57-59cf204c6c27/volumes"
Oct 31 20:28:06.207503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:06.207473    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:28:10.685263 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:10.685229    5958 scope.go:122] "RemoveContainer" containerID="9fa8ae70e56788fe93c14cb4f0d71100f6351693fce00edf4bed3fa1a2e80dad"
Oct 31 20:28:14.589227 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:14.589185    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:28:17.099756 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:28:17.099719    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:28:17.364816 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:17.364696    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t5m2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:28:17.365131 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:17.365099    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t5m2j" podUID="f8557080-882a-4d3e-8dde-c1f13e43564b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:28:19.823123 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:19.823088    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:28:35.089654 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:28:35.089617    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:28:47.044586 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:28:47.044532    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:29:01.807571 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:01.807516    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9vnsw"]
Oct 31 20:29:10.710181 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:10.710147    5958 scope.go:122] "RemoveContainer" containerID="1e244da13bea1c63de3c42e5bf98de13f0c504ddc9bd993dc8726b37ff5f466b"
Oct 31 20:29:12.706673 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:12.706643    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5748ddb879-r6qth_f065ea02-d1f3-4ab5-812a-4527f89650a6/manager/0.log"
Oct 31 20:29:12.906610 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:12.906582    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hbjrc_c81050c1-9e46-4570-b1a5-c9faa5de4014/manager/0.log"
Oct 31 20:29:13.593511 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:13.593472    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8689db56c-5m4t5_d0bc7265-4705-42fc-9e36-6497af8933e3/manager/0.log"
Oct 31 20:29:14.036522 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:14.036483    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-v2zwj_a0bd8286-fc8a-46e4-9c81-ca98bd386d0d/manager/0.log"
Oct 31 20:29:14.253375 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:14.253338    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c79hl_7f56ac52-327e-4e63-a190-346728a6e78c/proxy/0.log"
Oct 31 20:29:15.202911 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:15.202875    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fd9fcc7c9-gpftj_560513c8-27f1-49d1-931d-c6a7c55fe344/maas-api/0.log"
Oct 31 20:29:15.570696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:15.570613    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-mgcvq_2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6/revoke-keys/0.log"
Oct 31 20:29:15.777789 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:15.777756    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-hw8bm_c50304b0-7a3a-4c95-a245-bcb5e2b95e6d/revoke-keys/0.log"
Oct 31 20:29:16.159601 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:16.159505    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-df8716d9-ft4sx_7080b189-6fdd-4106-8daa-7f52d6ad8b74/revoke-keys/0.log"
Oct 31 20:29:16.525768 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:16.525683    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-v4wx9_cf4ca2ac-f190-45de-a720-bc15b2fe094a/postgres/0.log"
Oct 31 20:29:17.045897 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:29:17.045861    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:29:19.105767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:19.105732    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9vnsw_d10c6766-069b-4096-9b7b-67ce81c55dd9/limitador/0.log"
Oct 31 20:29:20.279999 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:20.279957    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gzhhr_cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757/discovery/0.log"
Oct 31 20:29:20.468784 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:20.468756    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-ff72l_9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43/kube-auth-proxy/0.log"
Oct 31 20:29:21.192488 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:21.192448    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-dh8nw_d2a89423-32aa-4b0c-a348-9eb35746f5b6/istio-proxy/0.log"
Oct 31 20:29:21.767070 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:21.767039    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-pdvwj_f4d48a85-41c3-4baf-9bcd-1bc0af46856d/istio-proxy/0.log"
Oct 31 20:29:22.299532 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:22.299502    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t5m2j_f8557080-882a-4d3e-8dde-c1f13e43564b/istio-proxy/0.log"
Oct 31 20:29:22.860811 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:22.860771    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-htgzs_8eb1e811-0d07-4067-ab2f-0c75b6fd2b70/payload-pre-processing/0.log"
Oct 31 20:29:25.095659 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:25.095631    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ggpfz_c104a69f-80bd-47cc-a6f0-b5e6e199e9c9/main/0.log"
Oct 31 20:29:25.279417 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:25.279390    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6_3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc/main/0.log"
Oct 31 20:29:35.442593 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:35.442553    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5748ddb879-r6qth_f065ea02-d1f3-4ab5-812a-4527f89650a6/manager/0.log"
Oct 31 20:29:35.652176 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:35.652141    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hbjrc_c81050c1-9e46-4570-b1a5-c9faa5de4014/manager/0.log"
Oct 31 20:29:36.273326 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:36.273285    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8689db56c-5m4t5_d0bc7265-4705-42fc-9e36-6497af8933e3/manager/0.log"
Oct 31 20:29:36.671023 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:36.670988    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-v2zwj_a0bd8286-fc8a-46e4-9c81-ca98bd386d0d/manager/0.log"
Oct 31 20:29:36.898604 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:36.898569    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-c79hl_7f56ac52-327e-4e63-a190-346728a6e78c/proxy/0.log"
Oct 31 20:29:37.648918 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:37.648885    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fd9fcc7c9-gpftj_560513c8-27f1-49d1-931d-c6a7c55fe344/maas-api/0.log"
Oct 31 20:29:38.030652 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:38.030568    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-mgcvq_2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6/revoke-keys/0.log"
Oct 31 20:29:38.214067 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:38.214028    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-hw8bm_c50304b0-7a3a-4c95-a245-bcb5e2b95e6d/revoke-keys/0.log"
Oct 31 20:29:38.586669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:38.586629    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-df8716d9-ft4sx_7080b189-6fdd-4106-8daa-7f52d6ad8b74/revoke-keys/0.log"
Oct 31 20:29:38.940818 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:38.940786    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-v4wx9_cf4ca2ac-f190-45de-a720-bc15b2fe094a/postgres/0.log"
Oct 31 20:29:40.027398 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:40.027348    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"]
Oct 31 20:29:40.029769 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:40.029675    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hw8bm"]
Oct 31 20:29:40.662395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:40.662358    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50304b0-7a3a-4c95-a245-bcb5e2b95e6d" path="/var/lib/kubelet/pods/c50304b0-7a3a-4c95-a245-bcb5e2b95e6d/volumes"
Oct 31 20:29:41.102073 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:41.101999    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9vnsw_d10c6766-069b-4096-9b7b-67ce81c55dd9/limitador/0.log"
Oct 31 20:29:42.039395 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:42.039358    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gzhhr_cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757/discovery/0.log"
Oct 31 20:29:42.225155 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:42.225126    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-ff72l_9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43/kube-auth-proxy/0.log"
Oct 31 20:29:42.958286 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:42.958256    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-dh8nw_d2a89423-32aa-4b0c-a348-9eb35746f5b6/istio-proxy/0.log"
Oct 31 20:29:43.514573 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:43.514517    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-pdvwj_f4d48a85-41c3-4baf-9bcd-1bc0af46856d/istio-proxy/0.log"
Oct 31 20:29:44.071917 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:44.071880    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t5m2j_f8557080-882a-4d3e-8dde-c1f13e43564b/istio-proxy/0.log"
Oct 31 20:29:44.624655 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:44.624624    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-htgzs_8eb1e811-0d07-4067-ab2f-0c75b6fd2b70/payload-pre-processing/0.log"
Oct 31 20:29:46.437314 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:46.437277    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ggpfz_c104a69f-80bd-47cc-a6f0-b5e6e199e9c9/main/0.log"
Oct 31 20:29:46.628990 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:46.628962    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-ghpj6_3cd8a8b1-d30c-440f-91f1-67f4f8ddd5bc/main/0.log"
Oct 31 20:29:47.045028 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:29:47.044989    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:29:56.027174 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:56.027133    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"]
Oct 31 20:29:56.036040 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:56.036000    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-mgcvq"]
Oct 31 20:29:56.663021 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:29:56.662986    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6" path="/var/lib/kubelet/pods/2bb92a6f-68ba-4e3c-a3da-ae2d5114a1f6/volumes"
Oct 31 20:30:16.026037 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:30:16.026000    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"]
Oct 31 20:30:16.028855 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:30:16.028829    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df8716d9-ft4sx"]
Oct 31 20:30:16.662475 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:30:16.662437    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7080b189-6fdd-4106-8daa-7f52d6ad8b74" path="/var/lib/kubelet/pods/7080b189-6fdd-4106-8daa-7f52d6ad8b74/volumes"
Oct 31 20:30:17.045139 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:30:17.045043    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:30:21.342531 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:30:21.342495    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8689db56c-5m4t5_d0bc7265-4705-42fc-9e36-6497af8933e3/manager/0.log"
Oct 31 20:30:21.621503 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:30:21.621473    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8689db56c-5m4t5_d0bc7265-4705-42fc-9e36-6497af8933e3/manager/0.log"
Oct 31 20:30:47.045082 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:30:47.044999    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:31:05.030421 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:05.030383    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8689db56c-5m4t5_d0bc7265-4705-42fc-9e36-6497af8933e3/manager/0.log"
Oct 31 20:31:05.410675 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:05.410637    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8689db56c-5m4t5_d0bc7265-4705-42fc-9e36-6497af8933e3/manager/0.log"
Oct 31 20:31:10.445820 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:10.445791    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bz76x_56c7eb65-3951-480e-88e2-1858cae684e6/global-pull-secret-syncer/0.log"
Oct 31 20:31:10.741514 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:10.741473    5958 scope.go:122] "RemoveContainer" containerID="d86c655b7d83b13082fca8ca87e643e6271f94e5a925ab4add76cf795de8f90f"
Oct 31 20:31:10.751107 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:10.751081    5958 scope.go:122] "RemoveContainer" containerID="6deff65dad2df46f1f9d058382a0a4e6959c6adf3145c44c9a46bcb68d682f0f"
Oct 31 20:31:10.781964 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:10.781937    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vgtp6_9a291037-a7e5-4cb8-bfde-a5aec293cbeb/konnectivity-agent/0.log"
Oct 31 20:31:10.894128 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:10.894098    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-35.us-east-2.compute.internal_a1d262f6c678a43aaab0402c233dbed3/haproxy/0.log"
Oct 31 20:31:17.043968 ip-10-0-138-35 kubenswrapper[5958]: E1031 20:31:17.043925    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx6db" containerName="extractor"
Oct 31 20:31:19.429871 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:19.429841    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9vnsw_d10c6766-069b-4096-9b7b-67ce81c55dd9/limitador/0.log"
Oct 31 20:31:20.572767 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.572740    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/alertmanager/0.log"
Oct 31 20:31:20.599336 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.599304    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"]
Oct 31 20:31:20.599827 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.599814    5958 cpu_manager.go:414] "containerMap: removing container" podUID="934df694-d4e6-4820-8a57-59cf204c6c27" containerName="payload-processing"
Oct 31 20:31:20.599882 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.599828    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934df694-d4e6-4820-8a57-59cf204c6c27" containerName="payload-processing"
Oct 31 20:31:20.599920 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.599903    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="934df694-d4e6-4820-8a57-59cf204c6c27" containerName="payload-processing"
Oct 31 20:31:20.603045 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.603029    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.611578 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.611534    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"]
Oct 31 20:31:20.615369 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.615346    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hk24f\"/\"kube-root-ca.crt\""
Oct 31 20:31:20.615369 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.615351    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hk24f\"/\"openshift-service-ca.crt\""
Oct 31 20:31:20.616728 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.616701    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/config-reloader/0.log"
Oct 31 20:31:20.645217 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.645180    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-podres\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.645412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.645252    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9hq\" (UniqueName: \"kubernetes.io/projected/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-kube-api-access-lh9hq\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.645412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.645354    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-sys\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.645412 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.645396    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-proc\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.645596 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.645463    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-lib-modules\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.654126 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.654106    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/kube-rbac-proxy-web/0.log"
Oct 31 20:31:20.686075 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.686043    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/kube-rbac-proxy/0.log"
Oct 31 20:31:20.709972 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.709939    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hk24f\"/\"default-dockercfg-pj4k7\""
Oct 31 20:31:20.732196 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.732165    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/kube-rbac-proxy-metric/0.log"
Oct 31 20:31:20.747039 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747007    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-proc\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747225 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747113    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-lib-modules\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747225 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747126    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-proc\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747225 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747190    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-podres\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747225 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747218    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-lib-modules\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747279    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-podres\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747289    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9hq\" (UniqueName: \"kubernetes.io/projected/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-kube-api-access-lh9hq\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747509 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747421    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-sys\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.747647 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.747518    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-sys\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.760868 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.760842    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9hq\" (UniqueName: \"kubernetes.io/projected/5fbb7f20-e5dc-47be-a018-c2f10e604c2b-kube-api-access-lh9hq\") pod \"perf-node-gather-daemonset-hh2s4\" (UID: \"5fbb7f20-e5dc-47be-a018-c2f10e604c2b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.763871 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.763844    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/prom-label-proxy/0.log"
Oct 31 20:31:20.794091 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.794064    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c5b8b8d7-f4d1-4fc4-aa0d-49780b322ee3/init-config-reloader/0.log"
Oct 31 20:31:20.885753 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.885721    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pnwjl_f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/kube-state-metrics/0.log"
Oct 31 20:31:20.910485 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.910455    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pnwjl_f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/kube-rbac-proxy-main/0.log"
Oct 31 20:31:20.919245 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.919214    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:20.944132 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.944106    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-pnwjl_f300c7e3-5d90-4bc7-bd34-9d8e743d9f27/kube-rbac-proxy-self/0.log"
Oct 31 20:31:20.987583 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:20.980892    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-674ffd47d7-5p6sd_98bdd4d0-f6c1-4a72-a603-e4928359958d/metrics-server/0.log"
Oct 31 20:31:21.175792 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.175706    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2kfm_555bb1b5-4eea-4de4-b4a0-45613823a90b/node-exporter/0.log"
Oct 31 20:31:21.197027 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.196993    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"]
Oct 31 20:31:21.198757 ip-10-0-138-35 kubenswrapper[5958]: W1031 20:31:21.198730    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5fbb7f20_e5dc_47be_a018_c2f10e604c2b.slice/crio-0bc676221c6d627d6576a9ce51e08c5e8842636a3aa4c743512722316c592e35 WatchSource:0}: Error finding container 0bc676221c6d627d6576a9ce51e08c5e8842636a3aa4c743512722316c592e35: Status 404 returned error can't find the container with id 0bc676221c6d627d6576a9ce51e08c5e8842636a3aa4c743512722316c592e35
Oct 31 20:31:21.205880 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.205861    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2kfm_555bb1b5-4eea-4de4-b4a0-45613823a90b/kube-rbac-proxy/0.log"
Oct 31 20:31:21.238903 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.238878    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l2kfm_555bb1b5-4eea-4de4-b4a0-45613823a90b/init-textfile/0.log"
Oct 31 20:31:21.375497 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.375467    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2zxqw_162fd1bc-c40c-4001-8fa3-8258ed577b45/kube-rbac-proxy-main/0.log"
Oct 31 20:31:21.409050 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.409020    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2zxqw_162fd1bc-c40c-4001-8fa3-8258ed577b45/kube-rbac-proxy-self/0.log"
Oct 31 20:31:21.437886 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.437811    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2zxqw_162fd1bc-c40c-4001-8fa3-8258ed577b45/openshift-state-metrics/0.log"
Oct 31 20:31:21.489696 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.489670    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/prometheus/0.log"
Oct 31 20:31:21.522420 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.522394    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/config-reloader/0.log"
Oct 31 20:31:21.551337 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.551307    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/thanos-sidecar/0.log"
Oct 31 20:31:21.581701 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.581671    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/kube-rbac-proxy-web/0.log"
Oct 31 20:31:21.612490 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.612452    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/kube-rbac-proxy/0.log"
Oct 31 20:31:21.632735 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.632697    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4" event={"ID":"5fbb7f20-e5dc-47be-a018-c2f10e604c2b","Type":"ContainerStarted","Data":"2913dde652e6a0b57970888ba524d6cb46c5c76e271d931f0c4eea3370095d46"}
Oct 31 20:31:21.632953 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.632778    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4" event={"ID":"5fbb7f20-e5dc-47be-a018-c2f10e604c2b","Type":"ContainerStarted","Data":"0bc676221c6d627d6576a9ce51e08c5e8842636a3aa4c743512722316c592e35"}
Oct 31 20:31:21.632953 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.632821    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:21.659218 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.659187    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/kube-rbac-proxy-thanos/0.log"
Oct 31 20:31:21.660908 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.660872    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4" podStartSLOduration=1.6608612219999999 podStartE2EDuration="1.660861222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:21.658161302 +0000 UTC m=+4275.119535918" watchObservedRunningTime="2026-10-31 20:31:21.660861222 +0000 UTC m=+4275.122235836"
Oct 31 20:31:21.697059 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.696950    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e212a43f-dd93-4653-ae12-87381b6fbb97/init-config-reloader/0.log"
Oct 31 20:31:21.741599 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.741566    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-t5chn_f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/prometheus-operator/0.log"
Oct 31 20:31:21.778752 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.778722    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-t5chn_f3e36838-e7e0-4a4d-8aa5-7324bb6fcbe2/kube-rbac-proxy/0.log"
Oct 31 20:31:21.857669 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.857625    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9b87db9-jxwtc_ec5f7e45-b28c-457d-afd8-92caec86aeb8/telemeter-client/0.log"
Oct 31 20:31:21.892948 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.892859    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9b87db9-jxwtc_ec5f7e45-b28c-457d-afd8-92caec86aeb8/reload/0.log"
Oct 31 20:31:21.923754 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.923721    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f9b87db9-jxwtc_ec5f7e45-b28c-457d-afd8-92caec86aeb8/kube-rbac-proxy/0.log"
Oct 31 20:31:21.960289 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.960183    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4c88497b-g2wmt_4184cf94-99b2-4491-a90f-a523b8f9ba3c/thanos-query/0.log"
Oct 31 20:31:21.990236 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:21.990194    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4c88497b-g2wmt_4184cf94-99b2-4491-a90f-a523b8f9ba3c/kube-rbac-proxy-web/0.log"
Oct 31 20:31:22.025682 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:22.025652    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4c88497b-g2wmt_4184cf94-99b2-4491-a90f-a523b8f9ba3c/kube-rbac-proxy/0.log"
Oct 31 20:31:22.057879 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:22.057851    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4c88497b-g2wmt_4184cf94-99b2-4491-a90f-a523b8f9ba3c/prom-label-proxy/0.log"
Oct 31 20:31:22.096671 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:22.096637    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4c88497b-g2wmt_4184cf94-99b2-4491-a90f-a523b8f9ba3c/kube-rbac-proxy-rules/0.log"
Oct 31 20:31:22.129834 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:22.129805    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b4c88497b-g2wmt_4184cf94-99b2-4491-a90f-a523b8f9ba3c/kube-rbac-proxy-metrics/0.log"
Oct 31 20:31:26.677222 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:26.677191    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vxmdl_d2d2e4e2-5258-4be1-929f-1d92b689c0ff/dns/0.log"
Oct 31 20:31:26.706442 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:26.706408    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vxmdl_d2d2e4e2-5258-4be1-929f-1d92b689c0ff/kube-rbac-proxy/0.log"
Oct 31 20:31:26.807394 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:26.807367    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w6g8x_d92bf693-c264-4e52-84a5-045804009257/dns-node-resolver/0.log"
Oct 31 20:31:27.557674 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:27.557641    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6qz8h_a237adff-6089-4d22-993f-95a9869a9771/node-ca/0.log"
Oct 31 20:31:27.648402 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:27.648371    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-hh2s4"
Oct 31 20:31:30.590623 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:30.590584    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gzhhr_cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757/discovery/0.log"
Oct 31 20:31:31.088086 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:31.088053    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-ff72l_9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43/kube-auth-proxy/0.log"
Oct 31 20:31:31.250031 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:31.250001    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-dh8nw_d2a89423-32aa-4b0c-a348-9eb35746f5b6/istio-proxy/0.log"
Oct 31 20:31:31.371068 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:31.371025    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-pdvwj_f4d48a85-41c3-4baf-9bcd-1bc0af46856d/istio-proxy/0.log"
Oct 31 20:31:31.492327 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:31.492295    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t5m2j_f8557080-882a-4d3e-8dde-c1f13e43564b/istio-proxy/0.log"
Oct 31 20:31:31.591383 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:31.591353    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-htgzs_8eb1e811-0d07-4067-ab2f-0c75b6fd2b70/payload-pre-processing/0.log"
Oct 31 20:31:32.218024 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:32.217994    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gzhhr_cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757/discovery/0.log"
Oct 31 20:31:32.230130 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:32.230106    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-ff72l_9c8ba7c4-8fd2-441f-9045-7e2bc9fdda43/kube-auth-proxy/0.log"
Oct 31 20:31:32.320823 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:32.320794    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-dh8nw_d2a89423-32aa-4b0c-a348-9eb35746f5b6/istio-proxy/0.log"
Oct 31 20:31:32.380661 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:32.380632    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-pdvwj_f4d48a85-41c3-4baf-9bcd-1bc0af46856d/istio-proxy/0.log"
Oct 31 20:31:32.440844 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:32.440808    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t5m2j_f8557080-882a-4d3e-8dde-c1f13e43564b/istio-proxy/0.log"
Oct 31 20:31:32.501592 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:32.501486    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-htgzs_8eb1e811-0d07-4067-ab2f-0c75b6fd2b70/payload-pre-processing/0.log"
Oct 31 20:31:34.505016 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:34.504975    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrq6l_39a2e073-9cf5-4472-9ba8-df174e1bb3e0/serve-healthcheck-canary/0.log"
Oct 31 20:31:35.376113 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:35.376082    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nx6db_31c31f22-7c5d-43bb-b554-ad463e4702c5/kube-rbac-proxy/0.log"
Oct 31 20:31:35.405657 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:35.405623    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nx6db_31c31f22-7c5d-43bb-b554-ad463e4702c5/exporter/0.log"
Oct 31 20:31:35.438161 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:35.438129    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nx6db_31c31f22-7c5d-43bb-b554-ad463e4702c5/extractor/0.log"
Oct 31 20:31:38.304981 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:38.304944    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bb8c5759f-dj7j4_94078d47-127e-41bc-9bad-4151d823f923/manager/0.log"
Oct 31 20:31:38.335736 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:38.335703    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bb8c5759f-dj7j4_94078d47-127e-41bc-9bad-4151d823f923/copy-manifests/0.log"
Oct 31 20:31:38.467719 ip-10-0-138-35 kubenswrapper[5958]: I1031 20:31:38.467685    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-gzhhr_cc0fa91f-68e3-4ba1-8cc7-e9a4d1704757/discovery/0.log"